Transmission terminal, transmission method, and recording medium storing transmission control program

ABSTRACT

A transmission terminal that communicates contents data including sound data with a counterpart transmission terminal, detects a delay time indicating a delay in receiving the contents data from the counterpart transmission terminal, and interrupts transmission of the contents data other than the sound data to the counterpart transmission terminal when the delay time exceeds a predetermined value.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is based on and claims priority pursuant to 35U.S.C. §119 to Japanese Patent Application No. 2012-227814, filed onOct. 15, 2012, in the Japan Patent Office, the entire disclosure ofwhich is hereby incorporated by reference herein.

BACKGROUND

Technical Field

The present invention generally relates to transmitting or receivingcontents data including sound data between a transmission terminal and acounterpart transmission terminal.

Description of the Related Art

The transmission systems allow transmission of contents data such asimage data and/or sound data among a plurality of transmission terminalsthat are remotely located from one another through a relay device tofacilitate communication among the plurality of transmission terminalsthrough a communication network such as the Internet. With the need forreducing costs or times associated with business trips, more companiesare moving towards transmission systems provided with theabove-described teleconference or videoconference capabilities.

Even in case there is a trouble in communication network or processingcapabilities of the transmission terminal, the transmission systems maystill transmit image data using the coding technique such as theH.264/Scalable Video Coding (SVC) as described in Japanese PatentApplication Publication No. 2010-506461-A. However, there may be somecases in which image data cannot be transmitted even with theabove-described coding technique, for example, when the transmissioncapability of the communication network is greatly lowered.

SUMMARY

Example embodiments of the present invention include a transmissionterminal, a transmission method, and a non-transitory recording mediumstoring a transmission control program, each of which communicatescontents data including sound data with a counterpart transmissionterminal, detects a delay time indicating a delay in receiving thecontents data from the counterpart transmission terminal, and interruptstransmission of the contents data other than the sound data to thecounterpart transmission terminal when the delay time exceeds apredetermined value.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of the disclosure and many of the attendantadvantages and features thereof can be readily obtained and understoodfrom the following detailed description with reference to theaccompanying drawings, wherein:

FIG. 1 is a schematic block diagram illustrating a configuration of atransmission system, according to an example embodiment of the presentinvention;

FIG. 2 is an illustration for explaining transmission or reception ofdata such as image data, sound data, or management data, performed bythe transmission system of FIG. 1;

FIGS. 3A to 3C are illustrations for explaining image quality of imagedata transmitted or received by the transmission system of FIG. 1;

FIG. 4 is a perspective view illustrating the outer appearance of atransmission terminal of the transmission system of FIG. 1;

FIG. 5 is a schematic block diagram illustrating a hardware structure ofthe transmission terminal of FIG. 1;

FIG. 6 is a schematic block diagram illustrating a hardware structure ofany one of the transmission management system, relay device, programproviding system, and maintenance system of the transmission system ofFIG. 1;

FIG. 7 is a schematic block diagram illustrating a functional structureof the transmission system of FIG. 1;

FIG. 8 is an example data structure of an image data management table,managed by the transmission terminal of FIG. 7;

FIG. 9 is a schematic block diagram illustrating a functional structureof a secondary selection unit of the transmission terminal of FIG. 7;

FIG. 10 is a schematic block diagram illustrating a functional structureof a primary selection unit of the transmission system of FIG. 7;

FIG. 11 is an example data structure of an image state change managementtable, managed by the relay device of FIG. 7;

FIG. 12 is an example data structure of a relay device management table,managed by the management system of FIG. 7;

FIG. 13 is an example data structure of a terminal authenticationmanagement table, managed by the management system of FIG. 7;

FIG. 14 is an example data structure of a terminal management table,managed by the management system of FIG. 7;

FIG. 15 is an example data structure of a candidate list managementtable, managed by the management system of FIG. 7;

FIG. 16 an example data structure of a session management table, managedby the management system of FIG. 7;

FIG. 17 is an example data structure of an address priority managementtable, managed by the management system of FIG. 7;

FIG. 18 is an example data structure of a transmission speed prioritymanagement table, managed by the management system of FIG. 7;

FIG. 19 is an example data structure of an image state management table,managed by the management system of FIG. 7;

FIG. 20 is data sequence diagram illustrating operation of managingstate information indicating an operation state of the relay device,according to an example embodiment of the present invention;

FIG. 21 is a data sequence diagram illustrating operation ofestablishing communication among two or more terminals of thetransmission system of FIG. 1, according to an example embodiment of thepresent invention;

FIG. 22 is a data sequence diagram illustrating operation of limiting anumber of candidate relay devices, performed by the transmission systemof FIG. 1 according to an example embodiment of the present invention;

FIG. 23 is a flowchart illustrating operation of limiting a number ofcandidate relay devices, performed by the management system of FIG. 7,according to an example embodiment of the present invention;

FIG. 24 is a table storing priority points of the relay devices that arerespectively calculated by the transmission management system of FIG. 7during the operation of limiting a number of candidate relay devices;

FIGS. 25A and 25B are a data sequence diagram illustrating operation ofselecting a relay device, performed by the transmission system of FIG.1, according to an example embodiment of the present invention;

FIG. 26 is a flowchart illustrating operation of selecting a relaydevice, performed by the terminal of FIG. 7, according to an exampleembodiment of the present invention;

FIG. 27 is a data sequence diagram illustrating operation oftransmitting or receiving data such as image data and sound data,performed by two or more terminals of the transmission system of FIG. 1,according to an example embodiment of the present invention;

FIG. 28 is an illustration of an example screen displayed through adisplay of the transmission terminal, according to an example embodimentof the present invention;

FIG. 29 is a flowchart illustrating operation of changing a screen onthe display in which image data is displayed, performed by thetransmission terminal of FIG. 7, according to an example embodiment ofthe present invention;

FIG. 30 is an example data structure of the image data management table,with the contents being changed by the transmission terminal of FIG. 7through operation of FIG. 29;

FIG. 31 is an illustration of an example screen displayed through thedisplay of the transmission terminal, based on the changed contents ofthe image data management table of FIG. 30;

FIG. 32 is a data sequence diagram illustrating operation of generating,transmitting, and displaying a message, performed by the transmissionsystem of FIG. 1 according to an example embodiment of the presentinvention;

FIG. 33 is a flowchart illustrating operation of generating a message,performed by the management system of FIG. 1, according to an exampleembodiment of the present invention;

FIG. 34 is a flowchart illustrating operation of displaying the message,performed by the transmission terminal of FIG. 1, according to anexample embodiment of the present invention;

FIG. 35 is an illustration of an example screen including the message,displayed through the display of the transmission terminal, according toan example embodiment of the present invention;

FIG. 36 is an illustration of an example screen including the message,displayed through the display of the transmission terminal, according toan example embodiment of the present invention;

FIG. 37 is an illustration of an example screen including the message,displayed through the display of the transmission terminal, according toan example embodiment of the present invention;

FIG. 38 is a flowchart illustrating operation of interruptingtransmission of image data, performed by the transmission terminal,according to an example embodiment of the present invention; and

FIG. 39 is a flowchart illustrating operation of interrupting capturingof image data, performed by the transmission terminal, according to anexample embodiment of the present invention.

The accompanying drawings are intended to depict example embodiments ofthe present invention and should not be interpreted to limit the scopethereof. The accompanying drawings are not to be considered as drawn toscale unless explicitly noted.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the presentinvention. As used herein, the singular forms “a”, “an” and “the” areintended to include the plural forms as well, unless the context clearlyindicates otherwise. It will be further understood that the terms“includes” and/or “including”, when used in this specification, specifythe presence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

In describing example embodiments shown in the drawings, specificterminology is employed for the sake of clarity. However, the presentdisclosure is not intended to be limited to the specific terminology soselected and it is to be understood that each specific element includesall technical equivalents that operate in a similar manner.

In the following description, illustrative embodiments will be describedwith reference to acts and symbolic representations of operations (e.g.,in the form of flowcharts) that may be implemented as program modules orfunctional processes including routines, programs, objects, components,data structures, etc., that perform particular tasks or implementparticular abstract data types and may be implemented using existinghardware at existing network elements or control nodes. Such existinghardware may include one or more Central Processing Units (CPUs),digital signal processors (DSPs),application-specific-integrated-circuits, field programmable gate arrays(FPGAs) computers or the like. These terms in general may be referred toas processors.

Unless specifically stated otherwise, or as is apparent from thediscussion, terms such as “processing” or “computing” or “calculating”or “determining” or “displaying” or the like, refer to the action andprocesses of a computer system, or similar electronic computing device,that manipulates and transforms data represented as physical, electronicquantities within the computer system's registers and memories intoother data similarly represented as physical quantities within thecomputer system memories or registers or other such information storage,transmission or display devices.

<Structure of Transmission System>

FIG. 1 is a schematic block diagram illustrating a transmission system,according to an example embodiment of the present invention. FIG. 2 isan illustration for explaining transmission or reception of data such asimage data, sound data, or management data, performed by thetransmission system of FIG. 1. FIGS. 3A to 3C are illustrations forexplaining image quality of image data transmitted or received by thetransmission system of FIG. 1.

In the following examples, it is assumed that the transmission system 1of FIG. 1 is implemented as the videoconference system, which is oneexample structure of the communication system. Based on this assumption,the transmission management system 50 is implemented as thevideoconference communication management system, which is one examplestructure of the communication management system. Further, thetransmission terminal 10 is implemented as the videoconferencecommunication terminal, which is one example structure of thecommunication terminal. However, the use of transmission system 1 is notlimited to the following examples such that the transmission system 1may be implemented as the transmission system or the communicationsystem.

More specifically, in the following examples, the user uses thetransmission terminal 10, such as the videoconference communicationterminal, to communicate with the user at the other end to havevideoconference.

The transmission system 1 of FIG. 1 includes a plurality of transmissionterminals 10 aa, 10 ab, 10 ac, 10 ba, 10 bb, 10 bc, 10 ca, 10 cb, 10 cd,10 da, 10 db, and 10 dc, and a plurality of displays 120 aa, 120 ab, 120ac, 120 ba, 120 bb, 120 bc, 120 ca, 120 cb, 120 cc, 120 da, 120 db, and120 dc, a plurality of relay devices 30 a, 30 b, 30 c, and 30 d, atransmission management system 50, a program providing system 90, and amaintenance system 100.

The transmission terminal 10 transmits or receives contents data such asimage data and/or sound data to or from another transmission terminal10.

For the descriptive purposes, in this example, any number of theplurality of terminals 10 aa to 10 dc may be collectively or eachreferred to as the terminal 10. Any number of the plurality of displays120 aa to 120 dc may be collectively or each referred to as the display120. Any number of the plurality of relay devices 30 a, 30 b, 30 c, and30 d may be collectively or each referred to as the relay device 30. Thetransmission management system 50 may be referred to as the “managementsystem” 50. Further, the terminal 10 that transmits data to anotherterminal 10 to carry out videoconference is referred to as the requestterminal 10. The terminal 10 that receives data from another terminal 10to carry out videoconference is referred to as the counterpart terminal10. For example, the request terminal 10 includes any terminal 10 thatrequests another terminal 10 to start videoconference, and thecounterpart terminal 10 includes any terminal 10 that is requested bythe request terminal 10 to start videoconference.

As illustrated in FIG. 2, in the transmission system 1, the requestterminal 10 and the counterpart terminal 10 first establish a managementdata session “sei” to start transmission and reception of various typesof management data through the management system 50. Further, in thisexample, the request terminal 10 and the counterpart terminal 10establish four contents data sessions sed to transmit or receivecontents data through the relay device 30. The four contents datasessions include a session “HL” to transmit high-level resolution imagedata HL, a session “ML” to transmit medium-level resolution image dataML, a session “LL” to transmit low-level resolution image data LL, and asession “V” to transmit sound data V such as voice data V. In thisexample, these four contents data sessions may be referred to as imageand/or sound data sessions.

Referring now to FIGS. 3A to 3C, various image data having differentresolution levels, which are respectively transmitted by the terminal 10of the transmission system 1, are explained. Referring to FIG. 3A, thelow-level resolution image data, which functions as a base image, has160 pixels in the horizontal direction and 120 pixels in the verticaldirection. Referring to FIG. 3B, the medium-level resolution image datahas 320 pixels in the horizontal direction and 240 pixels in thevertical direction. Referring to FIG. 3C, the high-level resolutionimage data has 640 pixels in the horizontal direction and 480 pixels inthe vertical direction. In case of communicating with a narrowbandsignal line, low-quality image data that is generated based on thelow-level resolution image data, which is the base image, istransmitted. In case of communicating with a wideband signal line,medium-quality image data that is generated based on the low-levelresolution image data and the medium-level resolution image data istransmitted. In case of communicating with a broadband signal line,high-quality image data that is generated based on the low-levelresolution image data, the medium-level resolution image data, and thehigh-level resolution image data is transmitted. Any one of theabove-described types of image data may be transmitted together withsound data such as voice data, as the sound data is relatively small interms of data size compared with that of image data.

Referring back to FIG. 1, the relay device 30 relays contents data suchas image data or sound data between or among the terminals 10. Forexample, the relay device 30, which may be alternatively referred to asthe relay device, may be implemented by a router or any device thatprovides the function of router. The management system 50 centrallymanages various information such as login information of the terminal10, the operation state (data transmission state or communication state)of the terminal 10, candidate list information, and the operation state(communication state) of the relay device 30. In this example, it isassumed that a moving image is transmitted as the image data.Alternatively, a still image, or both of the still image and the movingimage, may be transmitted as the image data.

The plurality of routers 70 a to 70 d, 70 ab, and 70 cd, which may becollectively or each referred to as the router 70, selects a route thatis most suitable for transmitting contents data such as image data andsound data.

The program providing system 90 includes a hard disk device (HD) 204(FIG. 6), which stores a terminal control program that causes theterminal 10 to perform various functions or operations. For example, theprogram providing system 90 sends the terminal control program to theterminal 10 through the Internet 2 i to cause the terminal 10 to installthe terminal control program. Further, the HD 204 of the programproviding system 90 may store a relay control program that causes therelay device 30 to perform various functions or operations. For example,the program providing system 90 sends the relay control program to therelay device 30 through the Internet 2 i to cause the relay device 30 toinstall the relay control program. Further, the HD 204 of the programproviding system 90 may store a transmission management program thatcauses the management system 50 to perform various functions oroperations. For example, the program providing system 90 sends thetransmission management program to the management system 50 to cause themanagement system 50 to install the transmission management program.

The maintenance system 100 is implemented as a computer capable ofmaintaining, managing, fixing, or upgrading at least one of the terminal10, relay device 30, management system 50, and program providing system90. Assuming that the maintenance system 100 is provided within acountry, and the terminal 10, relay device 30, management system 50, andprogram providing system 90 are each installed outside the country, themaintenance system 100 maintains, manages, fixes, or upgrades at leastone of the terminal 10, relay device 30, management system 30, andprogram providing system 90, remotely through the communication network2. The maintenance system 100 may manage maintenance of at least one ofthe terminal 10, relay device 30, management system 50, and programproviding system 90 without using the communication network 2. Forexample, a machine type number, a manufacturing number, customerinformation, maintenance and repair information, and failure loginformation may be maintained at the maintenance system 100 withoutusing the communication network 2.

Still referring to FIG. 1, the terminals 10 aa, 10 ab, and 10 ac, therelay device 30 a, and the router 70 a are connected to a local areanetwork (LAN) 2 a. The terminals 10 ba, 10 bb, and 10 bc, the relaydevice 30 b, and the router 70 b are connected to a LAN 2 b. The LAN 2 aand the LAN 2 b are connected to a leased line 2 ab in which the router70 ab is provided. It is assumed that these devices including theterminals 10 aa to 10 bc are located in an area A. For example, assumingthat the area is any area in Japan, the LAN 2 a could be located withinan office in a city such as Tokyo, and the LAN 2 b could be locatedwithin an office in another city such as Osaka.

The terminals 10 ca, 10 cb, and 10 cc, the relay device 30 c, and therouter 70 c are connected to a LAN 2 c. The terminals 10 da, 10 db, and10 dc, the relay device 30 d, and the router 70 d are connected to a LAN2 d. The LAN 2 c and the LAN 2 d are connected to a leased line 2 cd inwhich the router 70 cd is provided. It is assumed that these devicesincluding the terminals 10 ca to 10 dc are located in an area B apartfrom the area A. For example, assuming that the area is any area in theUnited States, the LAN 2 c could be located within an office in a citysuch as New York, and the LAN 2 d could be located within an office inanother city such as Washington, D.C. The area A and the area B areconnected through the Internet 2 i, via the routers 70 ab and 70 cd.

The management system 50 and the program providing system 90 areconnected through the Internet 2 i to the terminal 10 and the relaydevice 30. Any one of the management system 50 and the program providingsystem 90 may be located at any location within or outside any one ofthe area A and the area B.

In this example, the communication network 2 includes the LAN 2 a, LAN 2b, leased line 2 ab, Internet 2 i, leased line 2 cd, LAN 2 c, and LAN 2d. Any one or any portion of these lines or any other lines that may beincluded in the communication network 2 may be implemented as wirednetwork or wireless network such as Wireless Fidelity (WiFi) network orBluetooth (Registered Trademark) network.

As shown in FIG. 1, the terminal 10, the relay device 30, the managementsystem 50, the router 70, and the program providing system 90 are eachprovided with four digit numbers. These four digit numbers separated bydots are the simple expressions of IP addresses respectively assigned toany one of the devices shown in FIG. 1, each of which has a function ofcommunication device. For example, the IP address of the terminal 10 aais “1.2.1.3”. For simplicity, it is assumed that the IP address isexpressed in IPv4. Alternatively, the IP address may be expressed inIPv6.

Further, in this example, the terminal 10 may be communicated in variousways. For example, at least two different terminals 10 that are locatedat different rooms in the same office, or at least two differentterminals 10 that are located at different offices that are remotelylocated from one another, may communicate with one another. In anotherexample, at least two different terminals 10 that are located in thesame room may communicate with one another. In another example, oneterminal 10 that is located indoor and another terminal 10 that islocated outdoor, or at least two different terminals 10 that are bothlocated outdoor, may communicate with one another. When the terminal 10is located outdoor, the terminal 10 communicates with the other terminal10 through a wireless network such as a wireless network designed for amobile phone.

<Hardware Structure of Transmission System>

FIG. 4 is a perspective view illustrating the outer appearance of thetransmission terminal 10 of the transmission system of FIG. 1. In FIG.4, the longitudinal direction of the terminal 10 is referred to as xdirection. The direction orthogonal to the x direction, which is thehorizontal direction of the terminal 10, is referred to as the ydirection. The direction orthogonal to the x direction and the ydirection is referred to as the z direction.

As illustrated in FIG. 4, the terminal 10 includes a body 1100, an arm1200, and a camera housing 1300. The body 1100 includes a back side wall1110 having a plurality of air intake holes that are formed over thenearly entire surface of the intake surface of the back side wall 1110.The body 1100 further includes a front side wall 1120 provided with anexhaust surface 1121 having a plurality of exhaust holes over the nearlyentire surface of the exhaust surface 1121. When a cooling fan that isprovided within the body 1100 is driven, air flows in through the intakeholes of the intake surface and out through the exhaust holes of theexhaust surface 1121. The body 1100 further includes a right side wall1130 formed with a sound pickup hole 1131. Through the sound pickup hole1131, a microphone 114 (FIG. 5) of the terminal 10 is able to catchsounds such as human voice or any sound including noise.

The body 1100 has an operation panel 1150, which is provided at a frontsurface toward the right side wall 1130. The operation panel 1150includes a plurality of operation buttons 108 a to 108 e (“the operationbutton 108”), a power switch 109, an alarm lamp 119, and a plurality ofsound output holes 1151. Through the sound output holes 1151, a speaker115 (FIG. 5) of the terminal 10 is able to output sounds such as soundsgenerated based on human voice. The body 1100 further includes a holder1160, which is provided at the front surface toward the left side wall1140. The holder 1160, which has a concave shape, accommodates thereinthe arm 1200 and the camera housing 1300. The right side wall 1130 isfurther provided with a plurality of connection ports 1132 a to 1132 c(“connection ports 1132”). The connection ports 1132 allow electricalconnection to an external device through an outside device connectionI/F 118 (FIG. 5). The body 1100 further includes a left side wall 1140,which is provided with a connection port to connect the external display120 to the display I/F 117 through a cable 120 c.

The arm 1200 is fixed to the body 1100 via a torque hinge 1210. With thetorque hinge 1210, the arm 1200 can be rotated in directions of up anddown with respect to the body, while making a tilt angle θ1 of up to 135degrees. FIG. 4 illustrates the case where the tilt angle θ1 is 90degrees.

The camera housing 1300 incorporates therein the camera 112 (FIG. 5)that takes an image of an object. The object may be a part of a user,document, or a room where the terminal 10 is located. The camera housing1300 is provided with a torque hinge 1310. The camera housing 1300 isfixed to the arm 1200 through the torque hinge 1310. With the torquehinge 1310, the camera housing 1300 can be rotated with respect to thearm 1200, in the direction of up, down, right, and left, such that thecamera housing 1300 is kept at a desired position. More specifically,the camera housing 1300 can be rotated, while making a pan angle θ2 fromabout −180 degrees to 180 degrees in the direction right and left, and atilt angle θ3 that ranges from about −45 degrees to +45 degrees in thedirection of up and down. In FIG. 4, the pan angle θ2 and the tilt angleθ3 are each 0 degree.

The relay device 30, the management system 50, the program providingsystem 90, and the maintenance system 100 are each implemented by ageneral-purpose computer such as a personal computer or a servercomputer. For simplicity, explanation of the outer appearance of thecomputer is omitted.

FIG. 5 is a schematic block diagram illustrating a hardware structure ofthe transmission terminal 10. As illustrated in FIG. 5, the terminal 10includes a central processing unit (CPU) 101, a read only memory (ROM)102, a random access memory (RAM) 103, a flash memory 104, a solid statedrive (SSD) 105, a medium drive 107, the operation button 108, the powerswitch 109, a network interface (I/F) 111, the camera 112, an imagingelement interface (I/F) 113, the microphone 114, the speaker 115, asound input/output interface (I/O I/F) 116, the display interface (I/F)117, the outside device connection interface (I/F) 118, and an alarmlamp 119, which are electrically connected through a bus 110 such as anaddress bus or data bus. The CPU 101 controls entire operation of theterminal 10. The ROM 102 stores therein a control program for executionby the CPU 101, such as an initial program loader (IPL). The RAM 103functions as a work area of the CPU 101. The flash memory 104 storestherein various data such as the terminal control program, image data,or sound data such as voice data. The SSD 105 controls reading orwriting of various data with respect to the flash memory 104 undercontrol of the CPU 101. The medium drive 107 controls reading or writingof various data with respect to a removable recording medium 106 such asa flash memory. The operation button 108 allows the user to input a userinstruction, for example, by allowing the user to select a communicationdestination such as the counterpart terminal 10B. The power switch 109allows the user to switch on or off the power of the terminal 10. Thenetwork I/F 111 allows the terminal 10 to transmit data through thecommunication network 2.

The camera 112 takes an image of an object to obtain image data undercontrol of the CPU 101. The imaging element I/F 113 controls operationof the camera 112. The microphone 114 catches sounds such as voice ofthe user at the terminal 10. The speaker 115 outputs sounds such assounds generated based on voice of the user at the counterpart terminal10B. The sound I/O I/F 116 controls input or output of sound signalssuch as voice signals with respect to the microphone 114 and the speaker115 under control of the CPU 101. The display I/F 117 transmits imagedata to the display 120 under control of the CPU 101. The outside deviceconnection I/F 118 controls connection of the terminal 10 to varioustypes of outside device. The alarm lamp 119 generates notification whenan error is detected in the terminal 10.

The display 120 may be implemented by a liquid crystal display (LCD) oran organic light emitting display, which displays various data such asan image of an object or an operation icon. As illustrated in FIGS. 4and 5, the display 120 is connected to the display I/F 117 through thecable 120 c. The cable 120 c may be implemented by an analog RCB (VGA)signal cable, a component video cable, a high-definition multimediainterface (HDMI) signal cable, or a digital video interactive (DVI)signal cable.

The camera 112 includes a plurality of devices such as a lens system,and a solid-state image sensing device that photo-electrically convertsa light to generate an image of an object. For example, the solid-stateimage sensing device includes a complementary metal oxide semiconductor(CMOS) or a charge coupled device (CCD).

The outside device connection I/F 118 may be connected to an outsidedevice such as an external camera, external microphone, or externalspeaker through a universal serial bus (USB) cable that is connectedthrough the port 1132 of the body 1100 (FIG. 4). When the externalcamera is connected to the terminal 10, the CPU 101 causes the terminal10 to capture an image using the external camera, rather than the camera112 that is incorporated in the terminal 10. When the externalmicrophone or the external speaker is connected to the terminal 10, theCPU 101 causes the terminal 10 to use the external microphone or theexternal speaker in place of the incorporated microphone 114 or theincorporated speaker 115.

The recording medium 106, which can be freely attached to or detachedfrom the terminal 10, includes any desired type of recording medium. Inalternative to the flash memory 104, any nonvolatile memory that isreadable and writable under control of the CUP 101 may be used such asElectrically Erasable and Programmable ROM (EEPROM).

In this example, the terminal 10 is provided with the camera 112, themicrophone 114, and the speaker 115. Alternatively, the terminal 10 maynot be provided with any one of the camera 112, the microphone 114, andthe speaker 115. For example, the outside device connection I/F 118 mayconnect with the outside device that functions any one of the camera112, the microphone 114, and the speaker 115, to provide the functionsof the camera 112, the microphone 114, and the speaker 115.

The terminal control program may be written onto a recording medium thatis readable by a general-purpose computer such as the recording medium106 in any format that is installable or executable by a general-purposecomputer. Once the terminal control program is written onto therecording medium, the recording medium may be distributed. Further, theterminal control program may be stored in any desired memory other thanthe flash memory 104, such as the ROM 102.

FIG. 6 illustrates a hardware structure of the management system 50 ofFIG. 1. The management system 50 includes a CPU 201, a ROM 202, a RAM203, the HD 204, a hard disk drive (HDD) 205, a medium drive 207, adisplay 208, a network interface (I/F) 209, a keyboard 211, a mouse 212,and a CD-ROM drive 214, which are electrically connected through a bus210 such as an address bus or a data bus.

The CPU 201 controls entire operation of the management system 50. TheROM 202 stores a control program for execution by the CPU 201, such asan IPL. The RAM 203 functions as a work area of the CPU 201. The HD 204stores therein various data such as the transmission management program.The HDD 205 controls reading or writing of various data with respect tothe HD 204 under control of the CPU 201. The medium drive 207 controlsreading or writing of various data with respect to a removable recordingmedium 206 such as a flash memory. The display 208 displays various datasuch as a cursor, menu, window, character, or image. The network I/F 209allows the management system 50 to transmit data through thecommunication network 2. The keyboard 211 includes a plurality of keys,each of which is used for inputting a user instruction through acharacter, a numeral, or a symbol. The mouse 212 allows the user toinput a user instruction including, for example, selection or executionof a specific instruction, selection of an area to be processed, andinstruction of cursor movement. The CD-ROM drive 214 controls reading orwriting of various data with respect to a CD-ROM 213. In alternative tothe CD-ROM 213, any removable recording medium may be used.

The transmission management program may be written onto a recordingmedium that is readable by a general-purpose computer such as therecording medium 206 or the CD-ROM 213 in any format that is installableor executable by a general-purpose computer. Once the transmissionmanagement program is written onto the recording medium, the recordingmedium may be distributed. Further, the transmission management programmay be stored in any desired memory other than the HD 204, such as theROM 202.

The relay device 30 is substantially similar in hardware structure tothe management system 50 of FIG. 6, except for replacement of themanagement program with a relay device control program that is used forcontrolling the relay device 30. The relay device control program may bewritten onto a recording medium that is readable by a general-purposecomputer such as the recording medium 206 or the CD-ROM 213 in anyformat that is installable or executable by the general-purposecomputer. Once the relay device control program is written onto therecording medium, the recording medium may be distributed. Further, therelay device control program may be stored in any desired memory otherthan the HD 204, such as the ROM 202.

The program providing system 90 is substantially similar in hardwarestructure to the management system 50 of FIG. 6, except for replacementof the management program with a program providing program that is usedfor controlling the program providing system 90. The program providingprogram may be written onto a recording medium that is readable by ageneral-purpose computer such as the recording medium 206 or the CD-ROM213 in any format that is installable or executable by thegeneral-purpose computer. Once the program providing program is writtenonto the recording medium, the recording medium may be distributed.Further, the program providing program may be stored in any desiredmemory other than the HD 204, such as the ROM 202.

Other examples of removable recording medium, which may be used in placeof the CD-ROM 213, include, but not limited to, compact disc recordable(CD-R), digital versatile disk (DVD), and blue ray disc.

Next, a functional structure of the transmission system of FIG. 1 isexplained according to an example embodiment of the present invention.FIG. 7 is a schematic block diagram illustrating functional structuresof the transmission system 1. As illustrated in FIG. 7, the terminal 10,the relay device 30, and the management system 50 exchange data with oneanother through the communication network 2. In FIG. 7, the programproviding system 90 and the maintenance system 100 of FIG. 1 areomitted.

<Functional Structure of Terminal>

The terminal 10 includes a data transmitter/receiver 11, an operationinput 12, a login request 13, an imaging unit 14, a sound input 15 a, asound output 15 b, a secondary relay device selection unit 16, a displaycontrol 17, a delay detector 18, a memory control 19, an extractor 20, asound analyzer 21, and a changer 22. These units shown in FIG. 7correspond to a plurality of functions or functional modules, which areexecuted according to an instruction of the CPU 101 (FIG. 5) that isgenerated according to the terminal control program being loaded fromthe flash memory 104 onto the RAM 103. The terminal 10 further includesa memory 1000 that may be implemented by the flash memory 104 and theRAM 103 of FIG. 5.

(Image Data Management Table)

The memory 1000 stores an image data management database (DB) 1001,which is implemented by an image data management table of FIG. 8. Theimage data management table of FIG. 8 is used to manage an image screennumber for identifying each one of a plurality of image screens that canbe displayed on the display 120, in association with a terminal ID foridentifying the terminal 10 that sends image data being displayed on thedisplay 120 as a screen image in the specific image screen. The imagescreen number, which may be referred to as the screen number, is oneexample of display section identification information for identifying aspecific display section that can be displayed on the display 120. Theterminal ID of the terminal 10 is one example of terminal identificationinformation of the terminal 10. The image data management table of FIG.8 indicates that the screen numbers “1” to “8” will be respectivelyassigned to the image screens that can be displayed on the display 120.

The image data management table of FIG. 8 is used to further manage avirtual screen number for identifying each one of one or more virtualimage screens, in association with a terminal ID for identifying theterminal 10 that sends image data being assigned to the specific virtualimage screen. The image data that is transmitted from the terminal 10having the terminal ID, which is associated with the virtual imagescreen number, is not displayed on the display 120. For example, whenthere are more than eight images to be displayed on the display 120, thescreen number of the virtual image screen is assigned to any image datatransmitted from the terminal 10, for example, after respectivelyassigning the image screens with the terminal IDs of the terminals 10.The image data management table of FIG. 8 indicates that the screennumbers “R1” to “R3” will be respectively assigned to the virtual imagescreens that are not displayed on the display 120. Alternatively, thenumber of virtual screen images may be increased or decreased, dependingon the number of terminals 10 that are participating in videoconference.The virtual screen number is one example of virtual sectionidentification information for identifying a specific virtual sectionthat is not displayed on the display 120. For simplicity, the displaysection identification information and the virtual sectionidentification information may be collectively referred to as sectionidentification information.

Referring now to FIGS. 5 and 7, a functional structure of the terminal10 is explained according to an example embodiment of the presentinvention. More specifically, in this example, the operations orfunctions that are performed by the terminal 10, which include theoperations or functions performed by the units shown in FIG. 7, areperformed in relation to one or more hardware devices of the terminal 10that are shown in FIG. 5.

The data transmitter/receiver 11, which may be implemented by thenetwork I/F 111 (FIG. 5) under control of the CPU 101, transmits orreceives various data or information to or from another terminal,device, or system, through the communication network 2. In this example,the data transmitter/receiver 11 starts receiving the operation stateinformation that indicates the operation state of each candidatecounterpart terminal 10 from the management system 50, before startingcommunication with any counterpart terminal 10B. The operation state ofthe candidate terminal 10 indicates whether the candidate terminal 10 isonline or offline. When the terminal 10 is online, the operation stateof the candidate terminal 10 further indicates whether the candidateterminal 10 is available for communication (“communication OK”), thecandidate terminal 10 is having communication with the other terminal(“communicating”), the candidate terminal 10 is online but transmissionof image data is interrupted (“online, interrupt”), the candidateterminal 10 is on line but in mute state (“online, mute”), or thecandidate terminal 10 is online but in trouble or error (“online,communicating, trouble”). For example, when the cable 120 c isdisconnected from the terminal 10, the operation state of the candidateterminal 10 is assumed to be in the trouble state.

The operation input 12 receives a user instruction input by the userthrough the operation button 108 or the power switch 109 (FIG. 5), undercontrol of the instructions received from the CPU 101. For example, whenthe user selects “ON” using the power switch 109, the operation input 12receives a user instruction for turning the power on, and causes theterminal 10 to turn on the power.

The operations or functions of the login request 13 are performedaccording to an instruction received from the CPU 101 (FIG. 5). When thepower of the terminal 10 is turned on, the login request 13automatically causes the data transmitter/receiver 11 to send loginrequest information that requests the login process, and a current IPaddress of the terminal 10, to the management system 50 through thecommunication network 2. When the power of the terminal 10 is turned offaccording to a user instruction received from the user through the powerswitch 109, the login request 13 causes the data transmitter/receiver 11to send current operation state information of the terminal 10 to themanagement system 50, which indicates that the power of the terminal 10is turned off. After the operation state information is sent, theoperation input 12 turns off the power of the terminal 10. As theoperation state information of the terminal 10 is sent every time thepower is turned off, the management system 50 is able to know that theterminal 10 is off-line in realtime.

The operations or functions of the imaging unit 14 of the terminal 10are performed by the camera 112 and the imaging element I/F 113according to an instruction received from the CPU 101 (FIG. 5). Theimaging unit 14 takes an image of an object to output image data of theobject.

The operations or functions of the sound input 15 a of the terminal 10are performed by the sound I/O I/F 116 according to an instructionreceived from the CPU 101, in cooperation with the microphone 114. Afterthe microphone 114 converts voice of the user at the terminal 10 to avoice signal, the sound input 15 a inputs the sound signal in the formof sound data for further processing. The operations or functions of thesound output 15 b of the terminal 10 are performed by the sound I/O I/F116 according to an instruction received from the CPU 101, incooperation with the speaker 115. The sound output 15 b outputs a soundsignal of sound data that is received from the counterpart terminal 10through the speaker 115.

The secondary relay device selection unit 16 selects one of the relaydevices 30 that is suitable for communication to start videoconference.More specifically, according to an instruction received from the CPU 101(FIG. 5), the secondary relay device selection unit 16 performsselection of the relay device 30 using a counter 16 a, a calculator 16b, and a secondary selector 16 c, as illustrated in FIG. 9.

The counter 16 a obtains date and time information indicating the dateand time at which the data transmitter/receiver 11 of the terminal 10receives preparatory transmit information when the preparatory transmitinformation is transmitted from another terminal 10. The calculator 16 bcalculates a time period T between the time when the preparatoryinformation is transmitted by another terminal 10 and the time when thepreparatory information is received at the terminal 10, based on thedifference between the time and date information obtained by the counter16 a and time and date information included in the preparatory transmitinformation. The secondary selector 16 b selects one of the relaydevices 30 having the minimum value of the time period t calculated bythe calculator 16 b.

The operations or functions of the display control 17 of the terminal 10of FIG. 7 are performed by the display I/F 117 according to aninstruction received from the CPU 101.

The display control 17 controls transmit of image data, which isgenerated based on image data of different resolutions, to the display120. The display control 17 further causes the display 120 that isprovided for the request terminal 10 to display a candidate list beforethe request terminal 10 starts videoconference with a desiredcounterpart terminal 10. The display control 17 further causes thedisplay 120 to display a message, based on message data received fromthe management system 50, to the user.

The display control 17 further causes the display 120 to display imagesbased on image data transmitted from a plurality of counterpartterminals 10, on the respective screens through the display 120.Assuming that the display control 17 receives image data or messagedata, the display control 17 refers to the image data management tableof FIG. 8 to extract the screen number associated with the terminal IDof the terminal 10 that sends the image data or the message data, anddisplays an image or a message on the screen of the display 120identified by the extracted screen number based on the received imagedata or message data. The terminal ID of the terminal 10 may betransmitted from the terminal 10 together with the image data or themessage data for display. In case the extracted screen number is any oneof the numbers “R1” to “R3” corresponding to the virtual image screensthat are not displayed on the display 120, the display control 17 maydisplay a message based on the message data on a section other than theimage screens of the display 120, for example, in a message display areaP10 of FIG. 28.

The delay detector 18 receives information indicating a delay time msindicating a time period in which contents data such as image data orsound data sent through the relay device 30 from another terminal 10 isdelayed, according to an instruction received from the CPU 101 (FIG. 5).More specifically, the delay detector 18 calculates the delay time ms,by subtracting a frame rate of image data that is currently received,from the predetermined frame rate of 30 fps (frames per second).

The memory control 19 is implemented by the SSD 105 of FIG. 5 accordingto an instruction received from the CPU 101. The memory control 19stores various data in the memory 1000, or reads out various data fromthe memory 1000. The memory 1000 stores therein various data such asterminal identification (ID) information for identifying the terminal10, and a password for authenticating a user at the terminal 10. Thememory 1000 further stores therein image data and/or sound data receivedas the terminal 10 communicates with a counterpart terminal 10 such thatthe image data and/or sound data are overwritten. Before the image datais overwritten, an image generated based on the image data is displayedonto the display 120. Before the sound data is output, sounds generatedbased on the sound data is output through the speaker 150.

The extractor 20 may be implemented by instructions of the CPU 101 ofFIG. 5. In one example, when the data transmitter/receiver 11 receivesthe terminal ID of the terminal 10 that sends message data, preferablytogether with the message data, the extractor 20 searches the image datamanagement table of FIG. 8 using the terminal ID of the terminal 10 as asearch key to extract the screen number that corresponds to the terminalID.

The sound analyzer 21, which may be implemented by the instructions ofthe CPU 101 (FIG. 5), detects whether the received sound data has aspecific range of sound signal levels. The detection result may be usedto determine whether the sound data includes meaningful data such ashuman voices or any sounds other than noise. The detection of specificrange of sound signal levels may be performed using any desired knowntechnique such as the sound detection method described in YasuhisaNiimi, Speech Recognition, Information Science Lecture E-19-3, KyoritsuPublishing, Co., 1979, pp. 68-72, which is hereby incorporated herein byreference.

The changer 22 may be implemented by the instructions of the CPU 101(FIG. 5). When it is determined that the sound data has the specificrange of sound signal levels, the changer 22 obtains the terminal ID ofthe terminal 10 that is transmitted with the sound data, and updates theimage data management table of FIG. 8 so as to associate the screennumber with the obtained terminal ID of the terminal 10. The screennumber may be associated based on information previously defined by amaker or a seller of the terminal 10. The display control 17 refers tothe updated contents of the image data management table of FIG. 8 todisplay images that are respectively transmitted from the otherterminals 10 on the respective image screens of the display 120.

In this example, any one of the terminal ID of the terminal 10 and therelay device ID of the relay device 30 includes any type ofidentification information that can be expressed by any language,character, symbol, mark, or any combination of language, character,symbol, and mark.

<Functional Structure of Relay Device>

Now, a functional structure of the relay device 30 is explained. Therelay device 30 includes a data transmitter/receiver 31, a statedetector 32, an image state checker 33, a data quality manager 34, animage state changer 35, and a memory control 39. These units shown inFIG. 7 correspond to a plurality of functions or functional modules,which are executed according to an instruction of the CPU 201 (FIG. 6)that is generated according to the relay device control program beingloaded from the HD 204 onto the RAM 203. The relay device 30 furtherincludes a memory 3000 that may be implemented by the RAM 203 and/or theHD 204 (FIG. 6).

(Image State Change Management Table)

The memory 3000 includes an image state change management database (DB)3001, which stores an image state change management table illustrated inFIG. 11. The image state change management table of FIG. 11 stores anInternet protocol (IP) address of the counterpart terminal 10B to whichimage data is transmitted through the relay device 30, in associationwith image state information. The image state information indicatesquality of image data to be transmitted through the relay device 30 tothe counterpart terminal 10B, or whether to interrupt transmission ofthe image data to the counterpart terminal 10B.

<Functional Structure of Relay Device>

Next, a functional structure of the relay device 30 is explainedaccording to an example embodiment of the present invention. Morespecifically, in this example, the operations or functions that areperformed by the relay device 30, which include the operations orfunctions performed by the units shown in FIG. 7, are performed inrelation to one or more hardware devices of the relay device 30 that areshown in FIG. 6.

The data transmitter/receiver 31, which may be implemented by thenetwork I/F 209 (FIG. 6), transmits or receives various data orinformation to or from another terminal, device, or system, through thecommunication network 2, under control of instructions received from theCPU 201.

The state detector 32, which is implemented by the CPU 201 of FIG. 6,detects an operation state of the relay device 30. For example, theoperation state includes the on-line state (“online”), the off-linestate (“offline”), the communicating state (“communicating”), and theinterrupted state (“interrupted”). The on-line state is a state in whichthe relay device 30 is turned on and available for datatransmission/reception. The off-line state is a state in which the relaydevice 30 is not available for data transmission/reception, for example,as the power is not turned on.

The image state checker 33, which is implemented by the CPU 201 of FIG.6, searches the image state change management DB 3001 (FIG. 11) usingthe IP address of the counterpart terminal 10 as a search key to extractthe image state information regarding the quality of image data suitableto communication with the counterpart terminal 10 or information whethertransmission of the image data should be interrupted.

The data quality manager 34, which may be implemented by the CPU 201 ofFIG. 6, changes the contents of the image state change management DB3001 based on the image state information that is received from themanagement system 50, such as image quality information indicating thedesired quality of image data. For example, assuming that the requestterminal 10 aa having the terminal ID “01 aa” communicates with thecounterpart terminal 10 db having the terminal ID “01 db” to transmit orreceive high quality image data during videoconference, transmission ofimage data may delay for various reasons. For example, if a requestterminal 10 bb and a counterpart terminal 10 ca start videoconferenceover the communication network 2, transmission of image data from therequest terminal 10 aa to the counterpart terminal 10 db tends to slowdown due to the increase in traffic. In such case, the relay device 30changes the quality of image data to be transmitted from high imagequality to lower image quality. More specifically, the contents in theimage state change management DB 3001 is changed from high-level imagequality to medium-level image quality, based on the image stateinformation indicating the use of medium-level image quality.

The image state changer 35, which may be implemented by the CPU 201 ofFIG. 6, changes the quality of image data received from the requestterminal 10 to the quality of image data according to the contents ofthe image state change management DB 3001, such as the qualityinformation of the image data. Alternatively, the image state changer 35may interrupt transmission of the image data received from the requestterminal 10, according to the contents of the image state changemanagement DB 3001, such as information indicating interruption of imagedata.

The memory control 39 is implemented by the HDD 205 of FIG. 6 accordingto an instruction received from the CPU 201. The memory control 39stores various data in the memory 3000, or reads out various data fromthe memory 3000.

<Functional Structure of Management System>

The management system 50 includes a data transmitter/receiver 51, aterminal authenticator 52, a state manager 53, a terminal extractor 54,a terminal state obtainer 55, a primary relay device selection unit 56,a session manager 57, an image state determiner 58, a memory control 59,a delay time manager 60, and a message generator 61. These units shownin FIG. 7 correspond to a plurality of functions or functional modules,which are executed according to an instruction of the CPU 201 (FIG. 6)that is generated according to the transmission management program beingloaded from the HD 204 onto the RAM 203. The management system 50further includes a memory 5000, which may be implemented by the HD 204(FIG. 6).

(Relay Terminal Management Table)

The memory 5000 includes a relay device management database (DB) 5001,which stores therein a relay device management table of FIG. 12. Therelay device management table of FIG. 12 stores, for each relay deviceID of the terminal 10, the operation state of the relay device 30, thereceived date and time at which the management system 50 receives theoperation state information indicating the operation state of the relaydevice 30 from the relay device 30, the IP address of the relay device30, and the maximum data transmission speed of the relay device 30 inMbps. For example, for the relay device 30 a having the relay device ID“111 a”, the relay device management table indicates that the operationstate is “ON LINE”, the received date and time at which the managementsystem 50 receives the operation state information is “13:00 PM of11/10/2009”, the IP address of the relay device 30 a is “1.2.1.2”, andthe maximum data transmission speed of the relay device 30 a is 100Mbps.

(Terminal Authentication Management Table)

The memory 5000 further includes a terminal authentication managementdatabase (DB) 5002, which stores a terminal authentication managementtable of FIG. 12. The terminal authentication management table of FIG.12 stores a plurality of terminal IDs respectively assigned to theterminals 10 that are managed by the management system 50, inassociation with a plurality of passwords that are previously determinedfor the respective terminals 10. For example, referring to the terminalauthentication management table of FIG. 12, the terminal 10 aa havingthe terminal ID “01 aa” is assigned with the password “aaaa”.

(Terminal Management Table)

The memory 5000 further includes a terminal management database (DB)5003, which stores a terminal management table of FIG. 14. The terminalmanagement table of FIG. 14 stores, for each one of the terminal IDsassigned to the terminals 10, the terminal name to be used forcommunication with the terminal 10, the operation state of the terminal10, the received date and time at which the management system 50receives the login request information from the terminal 10, and the IPaddress of the terminal 10. For example, for the terminal 10 aa havingthe terminal ID “01 aa”, the terminal management table of FIG. 14indicates that the terminal name is “Japan Tokyo Office AA terminal”,the operation state is online (“ONLINE”) and is available forcommunication (“COMMUNICATION OK”), the received date and time is “13:40PM, 11/10/2009”, and the IP address of the terminal 10 aa is “1.2.1.3”.

(Candidate List Management Table)

The memory 5000 further includes a candidate list management database(DB) 5004, which stores a candidate list management table of FIG. 15.The candidate list management table of FIG. 15 stores, for each one of aplurality of request terminals 1 OA capable of requesting forvideoconference communication, the terminal ID of the request terminal10A, and one or more terminal IDs that are respectively assigned tocandidate terminals 10 that are previously registered for the requestterminal 10A. In this example, for the request terminal 10A, one or moreterminals 10 of the communication system 1 of FIG. 1 are previouslyregistered as the candidate terminal 10. For example, the candidate listmanagement table of FIG. 15 indicates that the request terminal 10 aahaving the terminal ID “01 aa” is most likely to request forvideoconference with respect to the terminal 10 ab having the terminalID “01 ab”, the terminal 10 ba having the terminal ID “01 ba”, and theterminal 10 bb having the terminal ID “01 bb”, etc. The managementsystem 50 manages the candidate list management table of FIG. 15, forexample, according to a user instruction received from any one of theterminals 10. For example, in response to a user instruction receivedfrom the terminal 10 aa, the management system 50 may add or delete thecontents of the candidate list management table of FIG. 15.

(Session Management Table)

The memory 5000 further includes a session management database (DB)5005, which stores a session management table of FIG. 16. The sessionmanagement table of FIG. 16 stores information regarding each of thesessions that are carried out by at least two terminals 10 of thetransmission system 1 for the purpose of selecting the relay device 30that is most suitable for communication between at least two terminals10. More specifically, for each session ID that uniquely identifies eachsession, the session management table of FIG. 16 stores a relay deviceID of the relay device 30 to be used for transmitting or receivingcontents data such as image data and sound data, a terminal ID of therequest terminal 10A, a terminal ID of the counterpart terminal 10B, adelay time ms indicating a time period required for receiving contentsdata at the counterpart terminal 10B, the date and time informationindicating the time at which the management system 50 receives delayinformation from the counterpart terminal 10B.

For example, referring to the session management table of FIG. 16, forthe session having the session ID “se1”, the relay device 30 a havingthe relay device ID “111 a” is selected to relay contents data betweenthe request terminal 10 aa having the terminal ID “01 aa” and thecounterpart terminal 10 db having the terminal ID “01 db”. Further, themanagement system 50 receives the delay information from the counterpartterminal 10 db at 14:00 PM, 11/10/2009. Based on the delay information,the delay time ms of 200 milliseconds (ms) is obtained. In case ofhaving videoconference between only two terminals 10, the time at whichthe delay time is received may be determined based on the time when themanagement system 50 receives the delay information transmitted from therequest terminal 10A rather than based on the time when the managementsystem 50 receives the delay information transmitted from thecounterpart terminal 10B. In case of having videoconference with morethan two terminals 20, the delay information transmitted from thecounterpart terminal 10 that receives the contents data is used tomanage the date and time at which the delay information is received.

(Address Priority Management Table)

The memory 5000 further includes a priority management database (DB)5006, which stores an address priority management table of FIG. 17. Theaddress priority management table of FIG. 17 defines a number of addresspriority points to be assigned to an arbitrary set of terminal 10 andrelay device 30 based on the degree of similarity between the IP addressof the terminal 10 and the IP address of the relay device 30. Assumingthat the IP address of the terminal 10 and the IP address of the relaydevice 30 are each expressed in the form of four digital numbers asdescribed above referring to FIG. 1, as the degree of similarity betweenthe terminal IP address and the relay device IP address increases, alarger number of address priority points is assigned. In FIG. 17, the“S” indicates that one digit of the IP address, which may be referred toas the dot address, is the same for both of the terminal 10 and therelay device 30. The “D” indicates that one digit of the IP address, orthe dot address, is different between the terminal 10 and the relaydevice 30.

More specifically, in this example, when the first to third digits ordot addresses are the same between the terminal 10 and the relay device30, the address priority point is 5. When the first and second digits ordot addresses are the same between the terminal 10 and the relay device30, the address priority point is 3. In such case, the fourth digit ordot address does not affect the address priority point. When the firstdigit or dot address is the same between the terminal 10 and the relaydevice 30, the address priority point is 1. In such case, the third andfourth digits or dot addresses do not affect the address priority point.When the first digit or dot address is different between the terminal 10and the relay device 30, the address priority point is 0. In such case,the second to fourth digits or dot addresses do not affect the addresspriority point.

(Transmission Speed Priority Management Table)

The priority management DB 5006 of the memory 5000 further includes atransmission speed priority management table of FIG. 18. Thetransmission speed priority management table of FIG. 18 stores a rangeof the maximum data transmission speeds in association with atransmission speed priority point. More specifically, the transmissionspeed priority management table of FIG. 18 indicates that thetransmission speed priority point increases with the increase in valueof the maximum data transmission speeds at the relay device 30. Forexample, referring to FIG. 18, when the maximum data transmission speedat the relay device 30 is equal to or greater than 1000 Mbps, thetransmission speed priority point of 5 is assigned. For example, whenthe maximum data transmission speed at the relay device 30 is equal toor greater than 100 Mbps but less than 1000 Mbps, the transmission speedpriority point of 3 is assigned. When the maximum data transmissionspeed at the relay device 30 is equal to or greater than 10 Mbps butless than 100 Mbps, the transmission speed priority point of 1 isassigned. When the maximum data transmission speed at the relay device30 is less than 10 Mbps, the transmission speed priority point of 0 isassigned.

(Image State Management Table)

The memory 5000 further includes an image state management database (DB)5007, which stores an image state management table of FIG. 19. The imagestate management table of FIG. 19 stores the delay time ms of image datain association with the image state information. More specifically, theimage data management table of FIG. 19 indicates that the quality ofimage data to be processed by the relay device 30 is lowered, as thedelay time ms of the image data at the request terminal 10A or thecounterpart terminal 10B increases. For example, when the delay time msis equal to or greater than 0 milliseconds (ms), but less than 100 ms,the image data quality is high. When the delay time ms is equal to orgreater than 100 ms but less than 300 ms, the image data quality ismedium. When the delay time ms is equal to or greater than 300 but lessthan 500 ms, the image data quality is low. When the delay time ms isequal to or greater than 500 ms, the management system 50 interruptsoperation of transmitting data.

(Functional Structure of Management System)

Next, a functional structure of the management system 50 is explainedaccording to an example embodiment of the present invention. In thisexample, the operations or functions that are performed by themanagement system 50, which include the operations or functionsperformed by the units shown in FIG. 7, are performed in relation to oneor more hardware devices of the management system 50 that are shown inFIG. 6.

The data transmitter/receiver 51, which may be implemented by thenetwork I/F 209 (FIG. 6) according to an instruction received from theCPU 201, transmits or receives various data or information to or fromanother terminal, device, or system through the communication network 2.

Under control of the CPU 201 (FIG. 6), the terminal authenticator 52obtains a terminal ID and a password from the login request informationthat is received from the data transmitter/receiver 51. Using theterminal ID and the password as a search key, the terminal authenticator52 searches the terminal authentication management DB 5002 (FIG. 12) todetermine whether the obtained set of terminal ID and password isregistered. Based on the search result, the terminal authenticator 52determines whether the user at the terminal 10 or the terminal 10 isallowed for access.

The state manager 53, which operates according to an instructionreceived from the CPU 201 (FIG. 6), manages the operation state of therequest terminal 10 that sends the login request information, using theterminal management DB 5003 (FIG. 14). The terminal management DB 5003stores therein the terminal ID of the request terminal 10A, theoperation state of the request terminal 10A, the date/time at which thelogin information is received at the management system 50, and the IPaddress of the request terminal 10A, in association with one another.For example, when the power of the terminal 10 is switched from the ONstate to the OFF state according to a user instruction received throughthe power switch 109, the state manager 53 receives the operation stateinformation of the terminal 10 indicating that the terminal 10 is turnedoff, from the terminal 10. Based on the operation state information ofthe terminal 10, the state manager 53 changes the operation stateinformation of the terminal 10 that is stored in the terminal managementDB 5003 from the on-line state to the off-line state.

The terminal extractor 54, which operates according to an instructionreceived from the CPU 201 (FIG. 6), searches the candidate listmanagement DB 5004 (FIG. 15) using the terminal ID of the requestterminal 10A that sends the login request information as a key to obtaina list of terminal IDs each being assigned to a plurality of candidateterminals 10 for the request terminal 10A. Additionally, the terminalextractor 54 searches the candidate list management DB 5004 (FIG. 15)using the terminal ID of the request terminal 10A that sends the loginrequest as a key to obtain a terminal ID of another request terminal 10Athat registers the request terminal 10A as a candidate terminal foranother request terminal 10A.

The terminal state obtainer 55, which operates under control of the CPU201 (FIG. 6), searches the terminal management DB 5003 (FIG. 14) usingthe terminal ID of each candidate terminal 10 that is extracted by theterminal extractor 54 as a key to obtain the operation state informationof each candidate terminal 10. More specifically, the terminal stateobtainer 55 obtains the operation state of each candidate counterpartterminal 10 that is previously registered as a candidate counterpartterminal for the request terminal 10A that sends the login requestinformation. Further, the terminal state obtainer 55 searches theterminal management DB 5003 using the terminal ID of the requestterminal 10A that is extracted by the terminal extractor 54 as a key toobtain the operation state information of the request terminal 10A thatsends the login request information. Further, the terminal stateobtainer 55 searches the terminal management DB 5003 using the terminalID of a candidate request terminal 10A that is extracted by the terminalextractor 54 as a key to obtain the operation state information of thecandidate request terminal 10A that lists the request terminal 10A thatsends the login request information as a candidate counterpart terminal.

The primary relay device selection unit 56, which operates according toan instruction received from the CPU 201 (FIG. 6), limits a number ofrelay devices 30 each of which is a candidate relay device 30 that maybe used for relaying contents data between at least two terminals 10.Based on the result obtained by the primary relay device selection unit56, the secondary relay device selection unit 16 of the terminal 10selects one terminal 30 that is most suitable for communication betweenat least two terminals 10. As illustrated in FIG. 10, the primary relaydevice selection unit 56 includes a session ID generator 56 a, aterminal IP address extractor 56 b, a primary selector 56 c, and apriority determiner 56 d.

The session ID generator 56 a of the primary relay device selection unit56 generates a session ID for identifying a session that is used forselecting the relay device 30. The terminal IP address extractor 56 bextracts the terminal ID of the request terminal 10A and the terminal IDof the counterpart terminal 10B respectively from the session requestinformation received from the request terminal 10A, and searches theterminal management DB 5003 (FIG. 14) to obtain the IP address of therequest terminal 10A and the IP address of the counterpart terminal 10B.The primary selector 56 c selects one or more relay devices 30 havingthe online state from the relay device management DB 5001 (FIG. 12) toobtain the relay device ID of the selected relay device 30. In thisexample, it is assumed that more than two relay devices 30 are selectedas having the on-line state.

Further, the primary selector 56 c obtains the IP address of each of theselected relay devices 30. Once the IP address of the relay device 30 isobtained for each relay device 30, the primary selector 56 c comparesthe IP address of the relay device 30 with at least one of the IPaddress of the request terminal 10A and the IP address of thecounterpart terminal 10B that are respectively obtained by the terminalIP address extractor 56 b to analyze the degree of similarity betweenthe IP address of the terminal 10 and the IP address of the relay device30. More specifically, the primary selector 56 c compares between the IPaddress of the terminal 10 and the IP address of the relay device 30,digit by digit, or dot address by dot address, to determine the degreeof similarity. Using the address priority management table of FIG. 17,the primary selector 56 c obtains the address priority point for eachone of the relay devices 30. Assuming that the primary selector 56 ccompares the IP address of the terminal 10 with the IP address of therelay device 30, respectively for the request terminal 10A and thecounterpart terminal 10B, the primary selector 56 c obtains two addresspriority points for each one of the relay devices 30. In such case, theprimary selector 56 c selects the highest one of the address prioritypoints as the address priority point for the relay device 30. For eachof the relay devices 30, the primary selector 56 c obtains a totalpriority point by adding the address priority point and the transmissionspeed priority point together. In this example, the primary selector 56c selects two relay devices 30 including the relay device 30 having thehighest total priority point and the relay device 30 having the secondhighest total priority point. In this example, a number of relay devices30 that is finally selected by the primary selector 56 c is not limitedto two such that more than two relay devices 30 may be finally selectedfor further processing as long as a number of relay devices 30 issufficiently reduced.

The priority determiner 56 d refers to the priority management DB 5006(FIG. 17) to determine the address priority point for each one of therelay devices 30 that is selected by the primary selector 56 c. Thepriority determiner 56 d obtains the maximum data transmission speed ofthe relay device 30 from the relay device management DB 5001 (FIG. 12),and refers to the priority management DB 5006 (FIG. 17) to obtain thetransmission speed priority point of the relay device 30 that isselected by the primary selector 56 c.

The session manager 57, which operates according to an instructionreceived from the CPU 201, stores the session ID generated by thesession ID generator 56 a, the terminal ID of the request terminal 10,and the terminal ID of the counterpart terminal 10, in a correspondingmanner, in the session management DB 5005 (FIG. 16) of the memory 5000.The session manager 57 further stores the relay device ID of the relaydevice 30 that is finally selected by the secondary selector 16 b of theterminal 10 for each session ID, in the session management DB 5005 (FIG.16).

The image state determiner 58, which operates according to aninstruction received from the CPU 201 (FIG. 6), searches the image statemanagement DB 5007 (FIG. 19) using the delay time ms obtained for theselected relay device 30 to obtain the image state information. Theimage state information is used to determine image quality that isdesirable for communication using the relay device 30 or to determinewhether transmission of the image data should be interrupted.

The memory control 59 is implemented by the HDD 205 of FIG. 6 accordingto an instruction received from the CPU 201. The memory control 59stores various data in the memory 5000, or reads out various data fromthe memory 5000.

The delay time manager 60 searches the terminal management DB 5003 (FIG.14) using the IP address of the counterpart terminal 10B to obtain theterminal ID of the counterpart terminal 10B. The delay time manager 60further manages the session management table of FIG. 16 stored in thesession management DB 5005 so as to keep updated the value stored in the“delay time” field for the obtained terminal ID of the counterpartterminal 10B.

The message generator 61 searches the image state management table (FIG.19) stored in the image state management DB 5007 using the delayinformation received at the data transmitter/receiver 51 as a search keyto obtain image state information that is associated with the receiveddelay information. When the image state information indicates thattransmission of image data should be interrupted, the message generator61 generates a message indicating that image data is not transmitted oronly sound data (voice data) is transmitted. More specifically, themessage generator 61 searches the terminal management DB 5003 (FIG. 14)using the address information that is received by the datatransmitter/receiver 51 together with the delay information as a searchkey to obtain a terminal ID of the terminal 10 having the receivedaddress information. The message generator 61 further searches thesession management DB 5005 (FIG. 16) using the terminal ID as a searchkey to obtain a terminal ID of the terminal having communication withthe terminal 10 that is specified. The message generator 61 furthersearches the terminal management DB 5003 (FIG. 14) using the terminal IDof the terminal 10 as a search key to obtain a terminal name of theterminal 10. The message generator 61 generates message data, whichincludes the terminal name of the terminal 10.

<Operation of Transmission System>

Referring now to FIGS. 20 to 27, example operation of the transmissionsystem 1 of FIG. 1 is explained. FIG. 20 is a data sequence diagramillustrating operation of managing state information indicating anoperation state of the relay device 30, which is transmitted from therelay device 30 to the management system 50. FIG. 21 is a data sequencediagram illustrating operation of establishing communication among twoor more terminals 10 of the transmission system of FIG. 1. FIG. 22 is adata sequence diagram illustrating operation of limiting a number ofcandidate relay devices. FIG. 23 is a flowchart illustrating operationof limiting a number of candidate relay devices. FIG. 24 is a tablestoring a calculation result of a priority point, which is used forlimiting a number of candidate relay devices 30. FIGS. 25A and 25B are adata sequence diagram illustrating operation of selecting the relaydevice 30, performed by the transmission system 1. FIG. 26 is aflowchart illustrating operation of selecting the relay device 30performed by the terminal 10. FIG. 27 is a data sequence diagramillustrating operation of transmitting or receiving data such as imagedata and sound data, performed by two or more terminals 10.

Referring now to FIG. 20, operation of managing state information of therelay device 30, which is sent from each relay device 30 to themanagement system 50, performed by the transmission system 1 isexplained according to an example embodiment of the present invention.In this example, it is assumed that the relay devices 30 a, 30 b, 30 c,and 30 d, which may be each or collectively referred to as the relaydevice 30, exist in the transmission system 1.

At S1-1, S1-2, S1-3, and S1-4, the relay devices 30 a, 30 b, 30 c, and30 d each periodically monitors the operation state of the relay device30. This monitoring is performed by the state detector 32 (FIG. 7) ofthe relay device 30.

At S2-1, S2-2, S2-3, and S2-4, the data transmitter/receiver 31 of therelay device 30 periodically transmits state information of the relaydevice 30 to the management system 50 through the communication network2. With the state information of the relay device 30 that isperiodically received, the management system 50 is able to manage theoperation state of the relay device 30 in realtime. The stateinformation of the relay device 30 includes an operation state of therelay device 30 that is detected by the state detector 32 of the relaydevice 30, which is sent together with a relay device ID that uniquelyidentifies each relay device 30. For the descriptive purposes, in thisexample, it is assumed that the relay devices 30 a, 30 b, and 30 d eachhave the on-line state, and the relay device 30 c has the off-line statedue to the failure in relay control program of the relay device 30 c.

At S3-1, S3-2, S3-3, and S3-4, the management system 50 receives thestate information from the relay device 30 at the datatransmitter/receiver 51, and stores the received state information ofthe relay device 30 in the memory 5000 through the memory control 59.More specifically, the memory control 59 stores the state information ofeach relay device 30 in association with the relay device ID of thecorresponding relay device 30 in the relay device management DB 5001(FIG. 12). For example, referring to FIG. 12, the management system 50stores the state information of the relay device 30 indicating whetherthe relay device 30 is on-line, off-line, or in trouble, etc., inassociation with the relay device ID of the relay device 30.Additionally, the management system 50 stores the date and timeinformation indicating the time when the management system 50 receivesthe state information of the relay device 30 in association with therelay device ID of the relay device 30. When the management system 50does not receive any state information from the relay device 30, therelay device management table of FIG. 12 has an empty value for the“operation state” field and the “date and time” field for the subjectedrelay device 30. Alternatively, the value of the “operation state” fieldand the value of the “date and time” field may reflect the stateinformation that is previously sent by the subjected relay device 30 tothe management system 50 such that the relay device management table ofFIG. 12 retains such value.

Referring now to FIG. 21, operation of transmitting and receivingvarious management data before starting videoconference with theterminal 10 db by the request terminal 10 aa is explained, according toan example embodiment of the present invention. In FIG. 21, managementdata is transmitted or received through the management data session sei.

For example, at S21, when the user selects “ON” using the power switch109, the operation input 12 receives a user instruction for turning thepower on, and causes the terminal 10 to turn on the power. At S22, asthe power of the request terminal 10 aa is turned on, the login request13 of the request terminal 10 aa automatically causes the datatransmitter/receiver 11 to send the login request information thatrequests the login process to the management system 50 through thecommunication network 2. The login request information includes aterminal ID that identifies the request terminal 10 aa, and a passwordassigned to the request terminal 10 aa. The terminal ID and the passwordmay be obtained by the memory control 19 from the memory 1000, and sentto the data transmitter/receiver 11. At the time of sending the loginrequest information from the request terminal 10 aa to the managementsystem 50, the request terminal 10 aa sends an IP address of the requestterminal 10 aa such that the management system 50 knows the IP addressof the request terminal 10 aa.

At S23, the terminal authenticator 52 of the management system 50searches the terminal authentication management DB 5002 (FIG. 12) storedin the memory 5000 using the terminal ID and the password of the loginrequest information received through the data transmitter/receiver 51.When it is determined that the terminal ID and the password of the loginrequest information is stored in the terminal authentication managementDB 5002, the terminal authenticator 52 determines that the terminal 10aa is authenticated.

At S24, when the terminal authenticator 52 authenticates that the loginrequest information is received from the authenticated terminal 10, thestate manager 53 of the management system 50 stores the operation state,the date and time at which the login request information is received,and the IP address of the terminal 10 aa, with respect to the terminalID in the terminal management DB 5003 (FIG. 14) to create a record ofthe terminal 10 aa. Using the terminal management table of FIG. 14,which stores the operations state of “online, communication OK”, thedate and time of “13:40, 11/10/2009”, and the terminal IP address of“1.2.1.3” in association with the terminal ID “01 aa”, variousinformation regarding the terminal 10 aa can be managed.

At S25, the data transmitter/receiver 51 of the management system 50sends the authentication result obtained by the terminal authenticator52 to the request terminal 10 aa that has sent the login requestinformation through the communication network 2. As described above, inthis example, it is assumed that the terminal authenticator 52determines that the terminal 10 aa is an authenticated terminal.

At S26, the terminal extractor 54 of the management system 50 searchesthe candidate list management DB 5004 (FIG. 15) using the terminal ID“01 aa” of the request terminal 10 aa that has sent the login requestinformation to extract a terminal ID and a terminal name for each ofcandidate terminals 10 that are previously registered for the requestterminal 10 aa. More specifically, referring to FIG. 15, the terminalextractor 54 extracts terminal IDs including “01 ab”, “01 ba”, “01 db”,etc. of terminals 10 ab, 10 ba, 10 db, etc., which are associated withthe request terminal 10 aa.

Further, at S27, the terminal state obtainer 55 searches the terminalmanagement DB 5003 (FIG. 14) using the extracted terminal IDs (“01 ab”,“01 ba”, and “01 db”, etc.) of the candidate terminals 10 extracted bythe extractor 54 as a search key to obtain the operation state for eachone of the candidate terminals 10 ab, 10 ba, 10 db, etc.

At S28, the data transmitter/receiver 51 of the management system 50sends the terminal ID (“01 ab”, “01 ba”, and “01 db”, etc.) and theoperation state information for each one of the candidate terminals 10ab, 10 ba, 10 db, etc. to the request terminal 10 aa through thecommunication network 2. More specifically, the datatransmitter/receiver 51 of the management system 50 sends the operationstate information of the candidate terminal 10 ab together with theterminal ID “01 ab” of the candidate terminal 10 ab as the terminalstate information. In this example, the terminal ID “01 ab” of thecandidate terminal 10 ab has been extracted by the terminal extractor54. Using the terminal ID “01 ab” as a search key, the operation stateof the candidate terminal 10 ab, which is the off-line state, isobtained. Similarly, the data transmitter/receiver 51 of the managementsystem 50 sends the terminal state information of the candidate terminal10 ba, which includes the terminal ID “01 ba” and the operation state ofthe candidate terminal 10 ba indicating the “online, interrupted” state.This process of sending the terminal state information is repeated untilthe terminal state information is sent for all of the candidateterminals 10. In this manner, the request terminal 10 aa is able toobtain the operation states (“offline”, “online”, and “online”, etc.) ofthe candidate terminals 10 ab, 10 ba, and 10 db, etc.

At S29, the terminal extractor 54 of the management system 50 searchesthe candidate list management DB 5004 (FIG. 15) using the terminal ID“01 aa” of the request terminal 10 aa that has sent the login requestinformation to extract the terminal ID of one or more terminals 10 eachof which registers the request terminal 10 aa as one of its candidateterminals 10. The candidate list management table of FIG. 15 indicatesthat the terminal ID of the terminal 10 having the request terminal 10aa as a candidate terminal is “01 ab”, “01 ba”, and “01 db”.

At S30, the terminal state manager 55 of the management system 50searches the terminal management DB 5003 (FIG. 14) using the terminal ID“01 aa” of the request terminal 10 aa that has sent the login requestinformation as a search key to obtain the operation state of the requestterminal 10 aa. In this example, the operation state “online,communication OK” is extracted for the request terminal 10 aa.

At S31-1 and S31-2, the data transmitter/receiver 51 of the managementsystem 50 sends the terminal ID “01 aa” and the operation state of therequest terminal 10 aa, which are respectively obtained at S30, to theterminals 10 ab, 10 ba, and 10 db each having the request terminal 10 aaas a candidate terminal that is obtained at S29. In this example, themanagement system 50 sends the terminal state information of the requestterminal 10 aa to only the terminals 10 ba and 10 db each having the“online, communication OK” state as shown in FIG. 14. Further, since theoperation state for the request terminal 10 aa is “online, communicationOK”, the operation state information “online, communication OK” istransmitted.

More specifically, in this example, the data transmitter/receiver 51refers to the terminal management table of FIG. 14 to obtain the IPaddress of each of the terminals 10 ba and 10 db. Using the obtained IPaddresses, the management system 50 is able to send the terminal stateinformation of the request terminal 10 aa to the terminals 10 ba and 10db each of which lists the request terminal 10 aa as a candidateterminal.

The above-described operation of S22 to S31-1 and 31-2 is performed byany desired terminal 10 as the power of the terminal 10 is turned onthrough the power switch 109 (FIG. 6).

Referring now to FIG. 22, operation of limiting a number of candidaterelay devices 30 is explained according to an example embodiment of thepresent invention. The operation of FIG. 22 is performed during amanagement data session sei (FIG. 2), which transmits or receivesvarious management data in the transmission system 1. Further, in thisexample, the request terminal 10 aa can start communication with atleast one of the terminals 10 ba, 10 db, etc., based on the terminalstate information received at S28 of FIG. 21. For the descriptivepurposes, it is assumed that the user at the request terminal 10 aastarts communication with the counterpart terminal 10 db.

At S41, the user at the request terminal 10 aa operates the operationbutton 108 to select the terminal 10 db as a counterpart terminal. Uponselection, the operation input 12 (FIG. 7) of the request terminal 10 aareceives a user instruction for starting communication with thecounterpart terminal 10 db.

At S42, the data transmitter/receiver 11 of the request terminal 10 aasends the communication start request information that requests themanagement system 50 to start communication with the counterpartterminal 10 db to the management system 50. The communication startrequest information at least includes identification information such asthe terminal ID “01 aa” of the request terminal 10 aa and the terminalID “01 db” of the counterpart terminal 10 db. With the communicationstart request information, the data transmitter/receiver 51 of themanagement system 50 receives the IP address “1.2.1.3” of the requestterminal 10 aa.

At S43, the state manager 53 looks for records in the terminalmanagement DB 5003 (FIG. 14) based on the terminal ID “01 aa” of therequest terminal 10 aa and the terminal ID “01 db” of the counterpartterminal 10 db, which are included in the communication start requestinformation. The state manager 53 changes each of the operation statesof the request terminal 10 aa and the counterpart terminal 10 db in therecords, from the online, communication OK state to the online,communicating state.

At this time, the request terminal 10 aa and the counterpart terminal 10db have not started communication, but the request terminal 10 aa andthe counterpart terminal 10 db each have the communicating state. Incase another terminal 10 tries to communicate with the request terminal10 aa or the counterpart terminal 10 db, the management system 50 causesthe another terminal 10 to output voice or display indicating that therequest terminal 10 aa or the counterpart terminal 10 db is in thecommunicating state.

At S44, the management system 50 prepares for a session that isperformed for selecting the relay device 30 for communication betweenthe request terminal 10 aa and the counterpart terminal 10 db. Morespecifically, at S44, the session ID generator 56 a (FIG. 10) of themanagement system 50 generates a session ID for a session that is to beperformed for selection of the relay device 30.

At S45, the session manager 57 stores the session ID “se1” generated atS44, the terminal ID “01 aa” of the request terminal 10 aa, and theterminal ID “01 db” of the counterpart terminal 10 db, in the sessionmanagement DB 5005 (FIG. 16) stored in the memory 5000.

At S46, the primary relay device selection unit 56 of the managementsystem 50 limits a number of candidate relay devices 30 from which onerelay device 30 to be used for communication between the requestterminal 10 aa and the counterpart terminal 10 db is selected, using therelay device management DB 5001, the terminal management DB 5003, andthe priority management DB 5006.

Referring now to FIG. 10 and FIG. 23, operation performed at S46 of FIG.22 is explained in detail. At S46-1 of FIG. 23, the terminal IP addressextractor 56 b of the management system 50 searches the terminalmanagement DB 5003 (FIG. 14) using the terminal ID “01 aa” of therequest terminal 10 aa and the terminal ID “01 db” of the counterpartterminal 10 db included in the communication start request informationsent from the request terminal 10 aa as a key to obtain the IP addressesof the terminals 10 aa and 10 db, i.e., the IP address “1.2.1.3” and theIP address “1.3.2.4”.

At S46-2, the primary selector 56 c refers to the relay devicemanagement DB 5001 (FIG. 12) to select one or more relay devices 30having the on-line operation state, and obtains the relay device ID ofthe selected relay device 30. More specifically, in this example, theprimary selector 56 c obtains the relay device IDs 111 a, 111 b, and 111d of the relay devices 30 a, 30 b, and 30 d.

At S46-3, the primary selector 56 c searches the relay device managementDB 5001

(FIG. 12) to obtain the IP address of each of the relay devices 30 a, 30b, and 30 d, using the relay device IDs 111 a, 111 b, and 111 d obtainedat S46-2. Further, the primary selector 56 c compares each one of the IPaddresses “1.2.1.2”, “1.2.2.2”, and “1.3.2.2” of the relay devices 30 a,30 b, and 30 d, with each one of the IP addresses “1.2.1.3” and“1.3.2.4” obtained at S46-1, dot address by dot address, to determinethe degree of similarity between the relay device IP address and theterminal IP address.

At S46-4, the priority determiner 56 d refers to the priority managementDB 5006 (FIG. 17) to determine a value of address priority point foreach one of the relay devices 30 a, 30 b, and 30 d. In this example, asillustrated in FIG. 24, for each one of the relay devices 30 a, 30 b,and 30 d, the priority determiner 56 d obtains an address priority pointwith respect to the request terminal 10 aa and an address priority pointwith respect to the counterpart terminal 10 db. FIG. 24 illustrates atable storing a calculation result of a priority point, which is usedfor limiting a number of candidate relay devices 30. The table of FIG.24 stores an address priority point, a transmission speed prioritypoint, and a total priority point, for each one of the relay devices IDsof the relay devices 30. The address priority point includes a firstaddress priority point with respect to the request terminal 10 aa, and asecond address priority point with respect to the counterpart terminal10 db. The total priority point is obtained by adding the highest one ofthe first and second address priority points with the transmission speedpriority point.

In this example, based on comparison between the IP address “1.2.1.2” ofthe relay device 30 a and the IP address “1.2.1.3” of the requestterminal 10 aa, the degree of similarity is “S.S.S.D” such that theaddress priority point of 5 is obtained. Similarly, based on comparisonbetween the IP address “1.2.1.2” of the relay device 30 a and the IPaddress “1.3.2.4” of the counterpart terminal 10 db, the degree ofsimilarity is “S.D.D.D” such that the address priority point of 1 isobtained. Based on comparison between the IP address “1.2.2.2” of therelay device 30 b and the IP address “1.2.1.3” of the request terminal10 aa, the degree of similarity is “S.S.D.D” such that the addresspriority point of 3 is obtained. Similarly, based on comparison betweenthe IP address “1.2.2.2” of the relay device 30 b and the IP address“1.3.2.4” of the counterpart terminal 10 db, the degree of similarity is“S.D.S.D” such that the address priority point of 1 is obtained. Basedon comparison between the IP address “1.3.2.2” of the relay device 30 dand the IP address “1.2.1.3” of the request terminal 10 aa, the degreeof similarity is “S.D.D.D” such that the address priority point of 1 isobtained. Similarly, based on comparison between the IP address“1.3.2.2” of the relay device 30 a and the IP address “1.3.2.4” of thecounterpart terminal 10 db, the degree of similarity is “S.S.S.D” suchthat the address priority point of 5 is obtained.

Referring back to FIG. 23, at S46-5, the priority determiner 56 dsearches the priority management DB 5006 (FIG. 18) using the maximumdata transmission speed of the relay device 30 that is stored in therelay device management DB 5001 (FIG. 12) to determine a transmissionpriority point for each one of the relay devices 30 a, 30 b, and 30 dthat are selected at S46-2.

In this example, referring to FIG. 12 and FIG. 18, the relay device 30 ahaving the maximum data transmission speed of 100 Mbps is assigned withthe transmission priority point of 3. Similarly, the relay device 30 bhaving the maximum data transmission speed of 1000 Mbps is assigned withthe transmission priority point of 5. Similarly, the relay device 30 dhaving the maximum data transmission speed of 10 Mbps is assigned withthe transmission priority point of 1. Accordingly, the prioritydeterminer 56 d stores the transmission priority point for each one ofthe relay devices 30 a, 30 b, and 30 d in the table of FIG. 24.

At S46-6, for each one of the relay devices 30 a, 30 b, and 30 d, theprimary selector 56 c adds the highest one of the first and secondaddress priority points with the transmission speed priority point toobtain a total priority point. The primary selector 56 c selects thetotal of two relay devices 30 having the highest priority point. Forexample, the primary selector 56 c selects the relay device 30 havingthe highest total priority point and the relay device 30 having thesecond highest total priority point as a candidate relay device 30 forfurther processing. In this example, referring to FIG. 24, the relaydevices 30 a, 30 b, and 30 d having the relay device IDs 111 a, 111 b,and 111 d respectively have the total priority points of 8, 8, and 6.Accordingly, the primary selector 56 c selects the relay device 30 ahaving the relay device ID 111 a, and the relay device 30 b having therelay device ID 111 b.

After the operation of S46 illustrated in FIG. 22 completes, at S47 ofFIG. 22, the data transmitter/receiver 51 (FIG. 7) of the managementsystem 50 sends the relay device selection information to thecounterpart terminal 10 db through the communication network 2. Therelay device selection information includes a number of candidate relaydevices 30, which is “2”, the terminal ID “01 aa” of the requestterminal 10 aa, and the session ID “se1” for relay device selection.With this relay device selection information, the counterpart terminal10 db is able to obtain information including the number of candidaterelay devices 30, the request terminal 10 aa that requests forvideoconference, and the session ID “se1” of the session for relaydevice selection. In addition, the counterpart terminal 10 db obtainsthe IP address “1.1.1.2” of the management system 50 that has sent therelay device selection information.

At S48, the data transmitter/receiver 11 of the counterpart terminal 10db sends confirmation information indicating that the relay deviceselection information is received, to the management system 50 throughthe communication network 2, with the IP address of the counterpartterminal 10 db. The confirmation information includes the session ID“se1”. With this confirmation information, the management system 50 isable to know that the counterpart terminal 10 db is notified with thenumber of candidate relay devices 30 obtained during the session se1,and the IP address “1.3.2.4” of the counterpart terminal 10 db.

Referring now to FIGS. 25A, 25B (FIG. 25), and 26, operation ofselecting the relay device 30, performed by the counterpart terminal 10db, is explained according to an example embodiment of the presentinvention. The operation of FIG. 25 is performed during the managementdata session sei of FIG. 2, which transmits or receives variousmanagement data in the transmission system 1.

Before starting videoconference, at S61-1 and S61-2, the managementsystem 50 sends preparatory relay request information, respectively, tothe relay devices 30 a and 30 b, which are selected by the managementsystem 50 at S46 as candidate relay devices. The preparatory relayrequest information requests the relay device 30 to perform relayprocessing before starting the videoconference. More specifically, thepreparatory relay request information includes the session ID “se1”, theIP address “1.2.1.3” of the request terminal 10 aa, and the IP address“1.3.2.4” of the counterpart terminal 10 db, and is transmitted with theIP address of the management system 50. With this preparatory relayrequest information, the relay devices 30 a and 30 b are each able toobtain information including the session, the request terminal, thecounterpart terminal, and the IP address “1.1.1.2” of the managementsystem 50 that has sent the preparatory relay request information.

At S62-1 and S62-2, the relay devices 30 a and 30 b each cause the datatransmitter/receiver 31 to send preparatory transmit request informationto the request terminal 10 aa through the communication network 2. Thepreparatory transmit request information requests the request terminal10 aa to send preparatory transmit information including the PacketInternet Grouper (PING) to each one of the relay devices 30 a and 30 bbefore starting the videoconference. More specifically, the preparatorytransmit request information includes the session ID “se1”, and istransmitted with the IP addresses of the relay devices 30 a and 30 b.With this preparatory transmit request information, the request terminal10 aa is able to know that the preparatory transmit information is to besent during the session with the session ID “se1”, as well as the IPaddresses “1.2.1.2” and “1.2.2.2” of the relay devices 30 a and 30 b.

As described above, the management system 50 does not directly send theIP address of the counterpart terminal 10 db to the request terminal 10aa. Instead, as described above referring to S61-1 and S61-2, themanagement system 50 sends the IP address of the counterpart terminal 10db respectively to the relay device 30 a and the relay device 30 b. Asdescribed above referring to S62-1, the relay device 30 aa requests therequest terminal 10 aa to send the preparatory transmit information tothe relay device 30 aa. In this manner, the management system 50prevents the terminal 10 from obtaining the IP address of anotherterminal 10, thus improving the security.

At S63-1 and S63-2, the request terminal 10 aa causes the datatransmitter/receiver 11 to send the preparatory transmit information,respectively, to the relay devices 30 a and 30 b through thecommunication network 2. The preparatory transmit information is sent tothe counterpart terminal 10 db through each one of the relay devices 30a and 30 b before the contents data such as the image data and the sounddata is transmitted. By sending the preparatory transmit information inplace of the contents data, the management system 50 is able tocalculate a time period required for transmitting the contents data fromthe request terminal 10 aa to the counterpart terminal 10 db througheach one of the relay devices 30 a and 30 b. Further, the preparatorytransmit information includes PING information used for checking whetherthe request terminal 10 aa, the relay device 30 a or 30 b, and thecounterpart terminal 10 db are each connected to allow communication,the date and time of which the request terminal 10 aa sends thepreparatory transmit information, and the session ID “se1”. With thispreparatory transmit information, each of the relay devices 30 a and 30b knows that the preparatory transmit information is transmitted in thesession with the session ID “se1”, and the IP address “1.2.1.3” of therequest terminal 10 aa that has sent the preparatory transmitinformation.

At S64-1 and S64-2, the relay devices 30 a and 30 b each transmit thepreparatory transmit information to the counterpart terminal 10 dbhaving the IP address “1.3.2.4”, which is obtained from the preparatorytransmit information. With the preparatory transmit information, thecounterpart terminal 10 db is able to know that the preparatory transmitinformation is transmitted during the session with the session ID “se1”,and the IP addresses “1.2.1.2” and “1.2.2.2” of the relay devices 30 aand 30 b that respectively send the preparatory transmit information.

At S65, the secondary relay device selection unit 16 of the counterpartterminal 10 db selects one of the relay devices 30 a and 30 b to be usedfor videoconference, based on the preparatory transmit information.

Referring now to FIG. 9 and FIG. 26, operation of selecting the relaydevice 30 for videoconference, which is performed at S65 of FIG. 25, isexplained. At S65-1, the counter 16 a of the secondary relay deviceselection unit 16 (FIG. 9) obtains the date and time at which the datatransmitter/receiver 11 of the counterpart terminal 10 db receives thepreparatory transmit information for each one of the relay devices 30 aand 30 b.

At S65-2, the calculator 16 b calculates, for each one of the relaydevices 30 a and 30 b, a time period between the time when thepreparatory transmit information is transmitted by the request terminal10 aa and the time when the preparatory transmit information is receivedby the counterpart terminal 10 db. The date and time at which thepreparatory information is transmitted by the request terminal 10 aa isobtainable from the preparatory transmit information. The date and timeof which the preparatory transmit information is received at thecounterpart terminal 10 db is obtained by the counter 16 a.

At S65-3, the secondary selector 16 c determines whether all items ofpreparatory transmit information is received for all of candidate relaydevices, during the session with the session ID “se1”. In this example,the secondary selector 16 c counts a total number of items ofpreparatory transmit information that have been received, and compareswith the total number of candidate relay devices 30 of “2”.

When it is determined that the preparatory transmit information has notbeen received for at least one relay device 30 (“NO” at S65-3), theoperation proceeds to S65-4. When it is determined that the preparatorytransmit information has been received for all of the candidate relaydevices 30 (“YES” at S65-3), the operation proceeds to S65-5. When it isdetermined that the predetermined time period has not passed (“NO” atS65-4), the operation returns to S65-1. When it is determined that thepredetermined time period has passed (“YES” at S65-4), the operationproceeds to S65-5.

At S65-4, the secondary selector 16 c determines whether a predeterminedtime period passes after the preparatory transmit information isreceived at the counterpart terminal 10 db. In this example, thepredetermined time period is set to one minute. When it is determinedthat the predetermined time period has not passed (“NO” at S65-4), theoperation returns to S65-1. When it is determined that the predeterminedtime period has passed (“YES” at S65-4), the operation proceeds toS65-5. At S65-5, the secondary selector 16 c selects one of the relaydevices 30, which has the least value of the time period required fortransmitting the preparatory transmit information based on thecalculation of the calculator 16 b.

In this example, it is assumed that the relay device 30 a is selected asa time period for transmitting the preparatory transmit information thatis relayed through the relay device 30 a has a value less than the valueof the time period for transmitting the preparatory transmit informationthat is relayed through the relay device 30 b.

Referring back to FIG. 25, at S66, the data transmitter/receiver 11 ofthe counterpart terminal 10 db sends the relay device selectioninformation to the management system 50 through the communicationnetwork 2. In this example, the relay device selection informationindicates that the relay device 30 a is selected. More specifically, therelay device selection information includes the session ID “se1”, andthe relay device ID “111 a” of the selected relay device 30 a, and istransmitted with the terminal IP address of the counterpart terminal 10db. With the relay device selection information, the management system50 is able to know that the relay device 30 a has been selected duringthe session with the session ID “se1”, and the IP address “1.3.2.4” ofthe counterpart terminal 10 db that has sent the relay device selectioninformation.

At S67, the session manager 57 of the management system 50 stores, inthe session management table of FIG. 16 stored in the session managementDB 5005, the relay device ID “111 a” of the relay device 30 a, which isfinally selected for communication, in the “relay device ID” field of arecord provided for the session with the session ID “se1”.

At S68, the data transmitter/receiver 51 of the management system 50sends the relay start request information to the relay device 30 athrough the communication network 2. The relay start request informationrequests the relay device 30 a to start relay operation. Morespecifically, the relay start request information includes the IPaddress “1.2.1.3” of the request terminal 10 aa, and the IP address“1.3.2.4” of the counterpart terminal 10 db.

At S69, the relay device 30 a establishes four sessions between therequest terminal 10 aa and the counterpart terminal 10 db including asession for transmission of low-level resolution image data, a sessionfor transmission of medium-level resolution image data, a session fortransmission of high-level resolution image data, and a session fortransmission of sound data. Once these sessions are established, therequest terminal 10 aa is able to start videoconference with thecounterpart terminal 10 db.

In the above-described example, the management system 50 sends the relaydevice selection information to the counterpart terminal 10 db at S47(FIG. 22), and the counterpart terminal 10 db performs operation of S48,S64-1 (FIG. 25A), S64-2 (FIG. 25B), and S65 (FIG. 25B) to select therelay device 30. In alternative to this example, the management system50 may send the relay device selection information to the requestterminal 10 aa to cause the request terminal 10 aa to perform selectionof the relay device 30. In such case, the request terminal 10 aaperforms operation of S48, S64-1, S64-2, and S65 in a substantiallysimilar manner as described above. Further, at S66, the request terminal10 aa sends the relay device selection information to the managementsystem 50.

Referring now to FIG. 7 and FIG. 27, operation of transmitting andreceiving contents data such as image data and sound data between therequest terminal and the counterpart terminal to carry outvideoconference, performed by the transmission system 1, is explainedaccording to an example embodiment of the present invention.

In this example, the contents data such as the image data and the sounddata flows in a direction from the request terminal 10 aa to thecounterpart terminal 10 db, or in another direction from the counterpartterminal 10 db to the request terminal 10 aa. Since operation such astransmission and reception of the contents data or detection of delaytime is the same for both of the directions, the following examplefocuses on communication in which data flows from the request terminal10 aa to the counterpart terminal 10 db via the relay device 30 a.

Referring to FIG. 27, at S81, the data transmitter/receiver 11 of therequest terminal 10 aa sends the contents data and the terminal ID “01aa” of the terminal 10 aa to the relay device 30 a through thecommunication network 2 in the contents data session “sed”. The contentsdata includes image data such as image data of an object captured by theimaging unit 14 and sound data that is input through the sound input 15a. In this example, it is assumed that the high-quality image data basedon the low-level resolution image data, the medium-level resolutionimage data, and the high-level resolution image data, and the sounddata, are transmitted. Accordingly, the data transmitter/receiver 31 ofthe relay device 30 a receives the image data of three differentresolution levels, and the sound data.

At S82, the image state checker 33 searches the image state changemanagement table (FIG. 11) using the IP address “1.3.2.4” of thecounterpart terminal 10 db as a key to obtain the image stateinformation, and to determine whether to change quality of image data tobe transmitted to the relay device 30 a or interrupt transmission ofimage data.

In this example, the quality of image data to be transmitted to therelay device 30 a is the high-quality image data. Since the image datathat is received at the data transmitter/receiver 31 has the qualitythat is the same as the quality of the image data obtained from theimage state change management DB 3001, at S83, the relay device 30 asends the high-quality image data and the sound data to the counterpartterminal 10 db in the contents data session “sed”, without applyingfurther image processing. The relay device 30 a further transmits theterminal ID “01 aa” of the request terminal 10 aa, to the counterpartterminal 10 db.

The counterpart terminal 10 db receives the high quality image data thatis generated based on the low-level resolution image data, medium-levelresolution image data, and high-level resolution image data, and thesound data, at the data transmitter/receiver 11. The display control 17combines the image data of three different resolution levels into thehigh quality image data for display onto the display 120. Further, thesound output 15 b outputs the sound based on the sound data.

At S84, the delay detector 18 of the counterpart terminal 10 dbperiodically receives information indicating a delay time indicating thetime at which the image data is received at the datatransmitter/receiver 11, for example, every one second. In this example,it is assumed that the delay time of 200 ms is obtained.

At S85, the data transmitter/receiver 11 of the counterpart terminal 10db sends the delay information indicating the delay time of 200 ms tothe management system 50 through the communication network 2, during themanagement data session “sei”. With the delay information, themanagement system 50 is notified of the delay time, and the IP address“1.3.2.4” of the counterpart terminal 10 db that has sent the delayinformation.

At S86, the delay time manager 60 of the management system 50 searchesthe terminal management table (FIG. 14) using the IP address “1.3.2.4”of the counterpart terminal 10 db as a search key to extract theterminal ID “01 db” of the counterpart terminal 10 db. The delay timemanager 60 stores the delay time of 200 ms obtained from the delayinformation in a “delay time” field of the record of the terminal ID “01db” of the session management table stored in the session management DB5005 (FIG. 16).

At S87, the image state determiner 58 searches the image statemanagement table stored in the image state management DB 5007 (FIG. 19)using the delay time of 200 ms to extract the image state information,that is, the image data quality of “MEDIUM”. Based on the extractedimage data quality, the image state determiner 58 determines that thequality of image data suitable for the delay time of 200 ms is medium.

At S88, the data transmitter/receiver 51 searches the relay devicemanagement table in the relay device management DB 5001 (FIG. 12) usingthe relay device ID “111 a”, which is stored in the session managementDB (FIG. 16) in association with the counterpart terminal ID “01 db”, toextract the IP address “1.2.1.2” of the relay device 30 a.

At S89, the data transmitter/receiver 51 sends the image stateinformation such as the quality information indicating that the imagedata quality that has been determined at S87 is medium-level, to therelay device 30 a through the communication network 2 through themanagement data session “sei”. The image state information istransmitted with the IP address “1.3.2.4” of the counterpart terminal 10db, which was used as a search key at S86.

At S90, the data quality manager 34 of the relay device 30 a stores theIP address “1.3.2.4” of the counterpart terminal 10 db in associationwith information regarding the “medium-level” quality of image data tobe relayed by the counterpart terminal 10 db, in the image state changemanagement DB 3001 (FIG. 11).

At S91, the request terminal 10 aa transmits the high quality image dataincluding the low-level resolution image data, the medium-levelresolution image data, and the high-level resolution image data, and thesound data, to the relay device 30 a through the contents data session“sed”, in a substantially similar manner as described above referring toS81.

At S92, the image state checker 33 of the relay device 30 a searches theimage state change management DB 3001 (FIG. 11) using the IP address“1.3.2.4” of the counterpart terminal 10 db as a search key to determinewhether to extract the quality of the image data suitable for thecounterpart terminal 10 db or to interrupt transmission of image data,in a substantially similar manner as described above referring to S82.

At S93, since the image data quality that is stored for the counterpartterminal 10 db is the medium-level, which is lower than the quality ofthe image data that is received at the data transmitter/receiver 31, theimage state changer 35 changes the quality of the image data from thehigh-level to the medium level. In this example, the quality of thesound data remains the same.

At S94, the data transmitter/receiver 31 of the relay device 30 sendsthe image data having the quality that is lowered to the medium-level,and the sound data, to the counterpart terminal 10 db through thecommunication network 2, during the contents data session “sed”.Further, the relay device 30 a sends the terminal ID “01 aa” of therequest terminal 10 aa to the terminal 10 db. The datatransmitter/receiver 11 of the counterpart terminal 10 db receives themedium-quality image data that is generated based on the low-levelresolution image data and the medium-level resolution image data, andthe sound data. The display control 17 of the counterpart terminal 10 dbcombines the image data of two different resolution levels to generatethe medium-level image data for display on the display 120. Further, thevoice output 15 b outputs the sounds generated based on the sound data.

As described above, when any delay in receiving the image data at thecounterpart terminal 10 db is observed, the relay device 30 a changesthe quality of image data by lowering the quality of image data.Accordingly, the users participating the videoconference are able tocarry out communication more smoothly.

Further, as long as the condition causing the delay is resolved, theimage quality is again changed from the lower image quality to thehigher image quality. For example, when the delay time is decreased to100 ms or lower as the loads on the communication network 2 decrease,after performing S85 to S92, at S93, the image quality is changed fromthe medium level to the high level.

Referring now to FIGS. 28 to 37, various image data to be displayed onthe display 120 db of the terminal 10 db is explained according to anexample embodiment of the present invention. In the following example,it is assumed that the terminal 10 aa, the terminal 10 ca, and theterminal 10 db are having videoconference, by sending contents datathrough the relay device 30 a as described above referring to FIG. 27.Further, in this example, the request terminal 10 transmits or receivescontents data to or from the counterpart terminal 10 through a contentsdata session assigned with a specific session ID, such that there arethree contents data sessions respectively for the terminal 10 db and theterminal 10 aa, the terminal 10 db and the terminal 10 ca, and theterminal 10 aa and the terminal 10 ca.

Accordingly, when viewed from the terminal 10 db, the terminal 10 dbtransmits or receives contents data with the terminal 10 aa through thecontents data session assigned with a specific session ID, and transmitsor receives contents data with the terminal 10 ca through the contentsdata session assigned with a specific session ID. For each contents datasession, it is assumed that data is transmitted or received as describedabove referring to FIG. 27.

FIG. 28 illustrates an example videoconference screen P0 being displayedon the display 120 db of the terminal 10 db. As illustrated in FIG. 28,the videoconference screen P0 displayed on the display 120 db includes amain screen P1 in which the image of one or more users at thecounterpart terminal 10 is displayed, a first sub-screen P2 in which theimage of one or more users at the terminal 10 at its own site isdisplayed, and a plurality of other sub-screens P3 to P8. In any one ofthe sub-screens P3 to P8, an image of one or more users participating invideoconference through the other terminal 10, other than the usersbeing displayed in the main screen P1 and the first sub-screen P2, maybe displayed. The main screen P1 is the largest in display size than theother sub-screens of the display 120. Further, in this example, the mainscreen P1 displays the site at which the user who is currently speakingis located. More specifically, in this example, the main screen P1displays an image at the terminal 10 aa or 10 ca. The sub-screen P2displays an image at the terminal 10 db. The sub-screen P3 displays animage at the terminal 10 aa or 10 ca.

The videoconference screen P0 further includes the message display areaP10 at the lower left portion of the screen P0, in which variousmessages are displayed. The videoconference screen P0 further includes a“SWITCH SCREEN” button P11 at the central portion of the screen P0. Whenselected such as a pointer using the mouse, the “SWITCH SCREEN” buttonP11 switches between an image displayed on the main screen P1 and one ofthe images displayed on the sub-screens P2 to P8.

The videoconference screen P0 further includes a camera icon P12 at thelower right portion of the screen P0. The camera icon P12 indicateswhether the camera 112 of the terminal 10 at its own site (in this case,the terminal 10 db) is turned on such that image data is transmitted orturned off such that image data is not transmitted. When the camera iconP12 is shown with the mark “X”, the camera 112 is turned on such thatthe user may turn off the camera 112 by selecting the camera icon P12.When the camera icon P12 is not shown with the mark “X”, the camera 112is turned off such that the user may turn on the camera 112 by selectingthe camera icon P12.

Referring now to FIGS. 29 to 31, operation of displaying an image of auser who is currently participating in videoconference through theterminal 10 on the main screen P1 of the display 120 is explainedaccording to a first example embodiment of the present invention. FIG.29 is a flowchart illustrating operation of changing an image beingdisplayed on a specific screen of the display 120. FIG. 30 is an exampledata structure of image data management table having the contents thatare changed. FIG. 31 illustrates an example screen displayed on thedisplay 120, after the screen is changed from the screen of FIG. 28based on the changed contents of the image data management table of FIG.30.

At S101 of FIG. 29, the sound analyzer 21 of the terminal 10, such asthe terminal 10 db, constantly analyzes sound data received at the datatransmitter/receiver 11 to determine whether the received sound dataincludes sounds having a sound signal level that is equal to or greaterthan a predetermined level. When it is determined that the sound datadoes not include sounds having a sound signal level equal to or greaterthan the predetermined level (“NO” at S101), the operation repeats S101.When it is determined that the sound data includes sounds having a soundsignal level equal to or greater than the predetermined level (“YES” atS101), the operation proceeds to S102.

In this example, it is assumed that the sound data received from thecounterpart terminal 10 ca includes sounds having a sound signal levelequal to or greater than the predetermined level.

At S102, the changer 22 updates information stored in the image datamanagement table stored in the image data management DB 1001, based onthe detection result at S101. More specifically, in this example, theimage data management table previously stores the terminal ID “01 aa” ofthe terminal 10 aa in association with the screen number “1”, theterminal ID “01 db” of the terminal 10 db in association with the screennumber “2”, and the terminal ID “ca” of the terminal 10 ca inassociation with the screen number “3”. With the information stored inthe image data management table, the display control 17 displays thescreen P0 as illustrated in FIG. 28. More specifically, the displaycontrol 17 displays the image of the terminal 10 aa in the main screenP1, the image of the terminal 10 db in the sub-screen P2, and the imageof the terminal 10 ca in the sub-screen P3.

Based on the detection result at S101, the changer 22 obtains theterminal ID “01 ca” of the terminal 10 ca that sends the sound datahaving a sound signal level equal to or greater than the predeterminedlevel, and associates the terminal ID “01 ca” with the screen number “1”as illustrated in FIG. 30. The changer 22 further associates theterminal ID “01 aa” with the screen number “3” as illustrated in FIG.30. As the information stored in the image data management table ischanged, the display control 17 displays images at different sites onthe respective screens P1, P2, and P3 on the display 120 db, asillustrated in FIG. 31. In FIG. 31, the image transmitted from theterminal 10 ca is displayed in the main screen P1, and the imagetransmitted from the terminal 10 aa is displayed on the sub-screen P3.

Through performing operation of FIG. 29, the display control 17 of theterminal 10 db displays an image showing a user who is currentlyspeaking on the main screen P1. As the user who is currently speaking isdisplayed in the larger screen, the participating user is able toinstantly recognize who is currently speaking.

In the above-described example referring to FIG. 27, at S85, it isassumed that the delay time of 200 ms is obtained, which indicates thatdelay in the time at which the image data transmitted from the terminal10 aa is received at the counterpart terminal 10 db. In such case, thequality of image data to be transmitted to the counterpart terminal 10db is made lower, thus increasing the transmission speeds. If the delaytime having much longer time is obtained from the counterpart terminal10 db, the management system 50 interrupts transmission of image datasuch that the terminal 10 db does not receive image data transmittedfrom the terminal 10 aa. In such case, the management system 50generates a message indicating that image data is not transmitted tonotify the user at the terminal 10 db. Alternatively, the managementsystem 50 may generate a message indicating that only sound data istransmitted to notify the user at the terminal 10 db.

More specifically, in the following example referring to FIGS. 32 to 36,it is assumed that the delay information transmitted from the terminal10 db through the contents data session with the terminal 10 aaindicates that the delay time is 600 ms, such that interruption isrequired. Further, it is assumed that the delay information transmittedfrom the terminal 10 db through the contents data session with theterminal 10 ca indicates that the delay time is less than 500 ms, suchthat interruption is not required.

In such case, in a similar manner as described above referring to S86,the delay time manager 60 of the management system 50 stores the delaytime of 600 ms obtained from the delay information received from thecounterpart terminal 10 db, in a “delay time” field of the record of theterminal ID “01 db” of the session management table stored in thesession management DB 5005 (FIG. 16).

Further, in a similar manner as described above referring to S87, theimage state determiner 58 of the management system 50 searches the imagestate management table stored in the image state management DB 5007(FIG. 19) using the delay time of 600 ms as a search key to extract theimage state information “INTERRUPT”. Based on the extracted image stateinformation “INTERRUPT”, the image state determiner 58 decides tointerrupt relaying of at least image data via the relay device 30 a.

As the image state information “INTERRUPT” is transmitted to the relaydevice 30 a that is currently relaying contents data, the relay device30 a stops relaying at least the image data. More specifically, at S94,the data transmitter/receiver 31 of the relay device 30 a transmits onlysound data, not image data, to the terminal 10 db through thecommunication network 2 in the contents data session “sed”.

When the image state information that is determined at S87 is“INTERRUPT” as described above, the management system 50 furtherperforms operation of generating a message as illustrated in FIG. 32 inconcurrently with S88 and S89. FIG. 32 illustrates a data sequencediagram illustrating operation of generating, transmitting, anddisplaying a message, performed by the transmission system of FIG. 1,according to an example embodiment of the present invention.

Referring to FIG. 32, at S121, the management system 50 generates amessage indicating that image data is not received at the terminal 10db, as described below referring to FIG. 33. FIG. 33 is a flowchartillustrating operation of generating a message, performed by themanagement system 50, according to an example embodiment of the presentinvention.

At S121-1 of FIG. 33, the message generator 61 of the management system50 searches the terminal management DB 5003 using the IP address of theterminal 10 db, which is received at S85, as a search key to obtain theterminal ID of the terminal 10 db.

At S121-2, the message generator 61 searches the session management DB5005 using the obtained terminal ID “01 db” of the terminal 10 db as asearch key to obtain the terminal ID “01 aa” of the terminal 10 aa,which is communicating with the terminal 10 db in the same sessionspecified by the session ID “se1”.

At S121-3, the message generator 61 searches the terminal management DB5003 using the terminal ID “01 aa” of the terminal 10 aa as a search keyto obtain the terminal name of the terminal 10 aa. When extracting theterminal name, any portion of the terminal name may be extracted. Forexample, “Japan Tokyo Office AA Terminal”, “Tokyo Office AA Terminal”,or “AA Terminal” may be extracted from the terminal name.

At S121-4, the message generator 61 combines a message indicating that“IMAGE NOT RECEIVED” with the extracted terminal name “TERMINAL AA” togenerate a message “IMAGE NOT RECEIVED FROM TERMINAL AA”. In alternativeto generating the message “IMAGE NOT RECEIVED”, the message generator 61may generate a message “ONLY SOUND IS RECEIVED” to indicate that onlythe sound data is received.

In alternative to generating the message in the form of text data, themessage may be generated in the form of image data such as an icon. Insuch case, image data such as an icon that reflects a specific messagemay be previously stored in the memory 5000. The message generator 61may read out image data such as the icon from the memory 5000 togenerate a message using the icon, and sends the generated message datato the terminal 10 db. Alternatively, the image data such as the iconmay be previously stored in the memory 1000 of the terminal 10 db.According to information received from the management system 50 thatrequests for displaying the icon reflecting that the image data is notdisplayed, the terminal 10 db may read out the image data such as theicon from the memory 1000 for display on the display 120 db.

Further, while the above-described example assumes that relaying of theimage data transmitted from the terminal 10 aa is stopped as delay isdetected, if relaying of the image data transmitted from the terminal 10ca is stopped as delay is detected, the management system 50 maygenerate message data indicating that image data transmitted from theterminal 10 ca is not received in a substantially similar manner asdescribed above referring to FIG. 33.

Referring back to FIG. 32, at S122, the data transmitter/receiver 51 ofthe management system 50 transmits the message data that is generated atS121 and the terminal ID “01 aa” of the terminal 10 aa the sends imagedata in which the delay is detected, to the terminal 10 db through thecommunication network 2. The data transmitter/receiver 11 of theterminal 10 db receives the message data and the terminal ID of theterminal 10 aa.

At S123, the display control 17 of the terminal 10 db causes the display120 db to display the message, which is received at the datatransmitter/receiver 11 of the terminal 10 db. In this example, themessage data is generated as text data, which is small in data sizecompared with image data or sound data, such that the message data canbe relayed through the communication network 2 even in the narrow band.

Referring now to FIGS. 34 and 35, operation performed at S123 isexplained in detail. FIG. 34 is a flowchart illustrating operation ofdisplaying a message based on message data received from the managementsystem 50, performed by the terminal 10 db, according to an exampleembodiment of the present invention. FIG. 35 illustrates an examplescreen displayed on the display 120 db, which includes the messagegenerated based on the message data.

At S123-1, the data transmitter/receiver 11 of the terminal 10 dbconstantly monitors the network to determine whether message data and aterminal ID are received at the data transmitter/receiver 11. When it isdetermined that the message data is not received (“NO” at S123-1), theoperation repeats S123-1. When it is determined that the datatransmitter/receiver 11 receives the message data and the terminal ID(“YES” at S123-1), the operation proceeds to S123-2. In this example, itis assumed that the message data and the terminal ID “01 aa” of theterminal 10 aa are received.

At S123-2, the extractor 20 of the terminal 10 db searches the imagedata management table of FIG. 8 using the received terminal ID as asearch key to extract the screen number that is associated with theterminal ID. In this example, it is assumed that the screen number “1”is associated with the terminal ID “01 aa”.

At S123-3, the display control 17 displays a message based on thereceived message data on the display 120 db. Assuming that the display120 db displays an image in the main screen P1 as illustrated in FIG.28, the display control 17 causes the display 120 db to display themessage in the main screen P1 as illustrated in FIG. 35. Morespecifically, assuming that the image data received from the terminal 10aa is displayed in the main screen P1 of the display 120 db at theterminal 10 db, and when the delay in receiving the image data from theterminal 10 aa is detected at the terminal 10 db for about 600 ms, theimage state changer 35 of the relay device 30 a interrupts relaying ofimage data received from the terminal 10 aa to the terminal 10 db. Asrelaying is stopped, the image data that is otherwise transmitted at S83or S94 of FIG. 27 is not transmitted to the terminal 10 db. In suchcase, as illustrated in FIG. 35, the terminal 10 db displays the messagein the main screen P1, which is identified using the terminal IDreceived at S122 of FIG. 32, based on the message data received at S122of FIG. 32.

Referring now to FIGS. 36 and 37, the other examples of displaying themessage in the screen of the display 120 db are explained. FIG. 36illustrates an example screen being displayed on the display 120 db,when delay is detected while the terminal 10 aa, the terminal 10 db, andthe terminal 10 ca are having videoconference. FIG. 37 illustrates anexample screen being displayed on the display 120 db, when delay isdetected while the terminals 10 including the terminals 10 aa, 10 db, 10ca, and 10 cc are having videoconference.

FIG. 36 illustrates the example case in which the management system 50determines to interrupt transmission of image data transmitted from theterminal 10 aa to the terminal 10 db, as the delay time of 600 ms isdetected at the terminal 10 db. In such case, the display control 17 ofthe terminal 10 db displays a message based on the message data receivedfrom the management system 50 at S122 of FIG. 32, in the sub-screen P3where the image data transmitted from the terminal 10 ca is previouslydisplayed. More specifically, referring back to S123-1 of FIG. 34, theterminal 10 db determines that the data transmitter/receiver 11 receivesthe message data and the terminal ID “01 aa” of the terminal 10 aa.

At S123-2, the extractor 20 of the terminal 10 db searches the imagedata management table of FIG. 8 using the received terminal ID as asearch key to extract the screen number “1” that is associated with theterminal ID “01 aa”.

In a substantially similar manner as described above referring to S102of FIG. 29, the changer 22 changes the contents stored in the image datamanagement table of FIG. 8 such that the screen number “1” is associatedwith the terminal ID of the terminal 10 other than the terminal 10 aa,that is, the terminal ID “01 ca” of the terminal 10 ca as illustrated inFIG. 30. The screen number “3” is associated with the terminal ID “01aa” of the terminal 10 aa.

At S123-3, the display control 17 of the terminal 10 db displays themessage based on the received message data on the sub-screen P3, asillustrated in FIG. 36. The display control 17 of the terminal 10 dbfurther displays the image based on the image data received from theterminal 10 ca on the main screen P1. The image of the terminal 10 dbcontinues to be displayed on the sub-screen P2.

In this manner, before displaying the message based on the message data,the terminal 10 db may switch between the image screen in which theimage data transmitted from the terminal 10 aa is currently displayed,and the image screen in which the image data transmitted from the otherterminal 10 ca is currently displayed.

For example, when it is determined that the image screen that isassociated with the terminal ID of the terminal 10 aa in which the delayis detected corresponds to the main screen P1, the terminal 10 db mayswitch the screen.

The screen of FIG. 37 is displayed when videoconference is carried outamong more than eight sites, i.e., more than eight terminals 10.Further, it is assumed that the participating terminals 10 include theterminal 10 aa, the terminal 10 db, the terminal 10 ca, the terminal 10cc, and five other terminals 10.

More specifically, in this example illustrated in FIG. 37, the imagedata transmitted from the terminal 10 aa, which is to be displayed inthe main screen P1 as illustrated in FIG. 28, is delayed in about 600ms. Further, it is assumed that the delay information transmitted fromeach one of the other terminals 10 through a contents data session foreach of the terminal 10 indicates that the delay time is less than 500ms, such that interruption is not required. In such case, the displaycontrol 17 of the terminal 10 db displays image data received from theterminal 10, which is otherwise not displayed in the screen P0, in themain screen P1 in place of the image of the terminal 10 aa. Further, themessage display screen P10 displays therein the message indicating thatthe image data is not received from the terminal 10 aa, based on themessage data received from the management system 50.

Referring back to FIG. 34, at S123-1, the terminal 10 db determines thatthe data transmitter/receiver 11 receives the message data and theterminal ID “01 aa” of the terminal 10 aa.

At S123-2, the extractor 20 of the terminal 10 db searches the imagedata management table of FIG. 8 using the received terminal ID as asearch key to extract the screen number “1” that is associated with theterminal ID “01 aa”. In this example, it is assumed that all of theimage screen numbers are associated with the terminal IDs of theterminals 10 that are currently participating, other than the terminal10 cc. Further, the virtual image screen number, for example, “R1” isassociated with the terminal ID of the terminal 10 cc.

In a substantially similar manner as described above referring to S102of FIG. 29, the changer 22 changes the contents stored in the image datamanagement table of FIG. 8 such that the screen number “1” is nowassociated with the terminal ID of the terminal 10 other than theterminal 10 aa, that is, the terminal ID “01 cc” of the terminal 10 cc.The virtual screen number “R1” is associated with the terminal ID “Olga”of the terminal 10 aa.

At S123-3, the display control 17 of the terminal 10 db displays themessage based on the received message data on the message display areaP10, as illustrated in FIG. 37. The display control 17 of the terminal10 db further displays the image based on the image data received fromthe terminal 10 cc on the main screen P1.

In this manner, before displaying the message based on the message data,the terminal 10 db may switch between the image screen in which theimage data transmitted from the terminal 10 aa is currently displayed,and the virtual image screen associated with the image data transmittedfrom the other terminal 10 cc.

In the above-described example, it is assumed that the image datacurrently displayed in the main screen P1 is not received as relaying isstopped, and that the main screen P1 is now associated with the imagedata of the terminal 10 cc that is not otherwise displayed, to displaythe image data of the terminal 10 cc in the main screen P1. In case theimage data currently displayed in any one of the sub-screens is notreceived as relaying is stopped, the sub-screen may be associated withthe image data of the terminal 10 cc that is not otherwise displayed, todisplay the image data of the terminal 10 cc in the sub-screen. In suchcase, the message data indicating that the image data is not received isdisplayed in the message display area P10.

As described above, in one example, when the relay device 30 stopsrelaying at least image data transmitted from the counterpart terminal10 to the terminal 10, the management system 50 generates and sendsmessage data indicating that image data is not received or only sounddata is received, to the terminal 10 that has been receiving the imagedata from the counterpart terminal 10 via the relay device 30. Themessage data may be sent together with the terminal ID of thecounterpart terminal 10. Based on the message data and the terminal IDreceived from the management system 50, the terminal 10 displays amessage based on the message data in a section of the display 120 thatis previously associated with the received terminal ID. With thismessage, a user at the terminal 10 knows that videoconference is notinterrupted, but image data is not received from the counterpartterminal 10 due to delay in data transmission. The user at the terminal10 is able to continue videoconference, for example, by responding tothe user at the counterpart terminal 10.

The management system 50 generates a message so as to include a name ofthe counterpart terminal 10 that transmits the image data. Withinformation indicating the terminal name of the counterpart terminal 10,the user at the terminal 10 is able to know that, from which terminalthe image data is not received, especially when videoconference takesplace among three or more terminals 10.

Further, when displaying the message at the terminal 10, the terminal 10may change a section that is supposed to display the message regardingthe counterpart terminal 10 in place of the image data that has beenpreviously received from the counterpart terminal 10, such that themessage is displayed in the other section.

For example, when the terminal 10 determines that the message is to bedisplayed in a main display section having relatively a large screensize, the terminal 10 may display the message in a section other thanthe main display section, by changing the association that associatesbetween a section and a terminal that sends image data for display onthe section.

In addition to the above, in one example, the terminal 10 furtherperforms operation of FIG. 38, thus reducing a data traffic that may becausing the delay in data transmission through the network. Morespecifically, when the delay time exceeds the predetermined value, suchas 500 ms, the delay detector 18 of the terminal 10 of FIG. 7 requeststhe data transmitter/receiver 11 to interrupt transmission of imagedata. Further, when the data transmitter/receiver 11 receives therequest for interruption of image data from the delay detector 18, thedata transmitter/receiver 11 transmits only sound data through thecommunication network 2, while interrupting transmission of image dataeven when the image data is being captured by the imaging unit 14.

Referring now to FIG. 38, operation of interrupting transmission ofimage data is explained according to an example embodiment of the presetinvention. In this example, it is assumed that operation from S81 to S84of FIG. 27 is performed to transmit contents data between the terminal10 aa and the terminal 10 db. After S84 of detecting the delay time, theterminal 10 db performs operation of FIG. 38, instead of performing S85to S94 of FIG. 27.

At S84 of FIG. 27, the delay detector 18 of the terminal 10 db detects adelay time indicating the delay in reception of contents data from theterminal 10 aa. When it is determined that the detected delay timeexceeds the predetermined value (such as 500 ms), (“YES” at S201), atS202, the delay detector 18 requests the data transmitter/receiver 11 tointerrupt transmission of image data.

At S203, the data transmitter/receiver 11 stops transmission of imagedata through the communication network 2, such that only sound data istransmitted from the terminal 10 db to the terminal 10 aa.

When it is determined that the detected delay time does not exceed thepredetermined value (“NO” at S201), the operation ends withoutperforming S202 and S203 as there is no need to interrupt transmissionof image data to resolve the problem in delay in data transmission. Insuch case, the contents data including image data and sound data istransmitted to the terminal 10 aa via the relay device 30 a.

In case the data transfer capability is lowered, for example, due to theincreased load on the communication network 2, the terminal 10 db totransmit image data simply stops transmission of image data. Thus, thereis not need to request the counterpart terminal 10 aa to interrupttransmission of image data, for example, via the management system 50,or there is no need for the management system 50 to manage variousinformation such as the delay time information or the image stateinformation. This easily reduces the data traffic as no image data istransmitted, thus contributing to the solution to the increased datatraffic on the communication network.

For example, in case there is a delay in data transmission, the imagedata may be transmitted while reducing the data size of the image datato be transmitted by lowering the image resolution. In some cases, thismay still result in interruption in transmission of image data, or mayeven result in interruption in transmission of sound data. In view ofthis, when the terminal 10 detects the delay in data transmission, theterminal 10, which transmits contents data to the counterpart terminal10, stops transmitting at least image data of the contents data thattends to have a large data size. Accordingly, at least sound data of thecontents data is transmitted to the counterpart terminal 10 to at leastkeep the flow of conversing voices of the users. Thus, as the sound datais transmitted to the counterpart terminal 10 aa, the users are able tocommunicate with each other to carry out teleconference.

Further, since the terminal 10 db stops transmission of image data, theterminal 10 db does not have to send a request to the counterpartterminal 10 aa to interrupt transmission of image data from thecounterpart terminal 10 aa to the terminal 10 db. In one example, whenthe counterpart terminal 10 aa detects a delay in reception of contentsdata, the terminal 10 aa performs operation of FIG. 38 in asubstantially similar manner as described above, thus transmitting onlysound data. In another example, when the counterpart terminal 10 aa doesnot detect a delay in reception of contents data, the terminal 10 aacontinues to transmit contents data including image data and sound data.

In another example, in addition to interrupting transmission of imagedata, the terminal 10 may further stop capturing the image, as there isnot need to transmit image data to the counterpart terminal 10.

FIG. 39 illustrates example operation of interrupting capturing of imagedata, which may be performed by the terminal 10 db, when the delay indata transmission is detected at S84 of FIG. 27.

When the delay detector 18 of the terminal 10 db of FIG. 7 detects adelay time indicating the delay in reception of contents data, whichexceeds the predetermined value (for example, 500 ms), the delaydetector 18 requests the imaging unit 14 to stop capturing an image ofthe object such as the user at the terminal 10 db. In response, theimaging unit 14 of the terminal 10 db stops capturing the image of theobject such as the user.

More specifically, it is assumed that S81 to S84 of FIG. 27 is performedto transmit contents data between the terminal 10 aa and the terminal 10db. After S84 of detecting the delay time, the terminal 10 db performsoperation of FIG. 39, instead of performing S85 to S94 of FIG. 27, inaddition to operation of FIG. 38 or in place of operation of FIG. 38.

At S84 of FIG. 27, the delay detector 18 of the terminal 10 db detects adelay time indicating the delay in reception of contents data from theterminal 10 aa. When it is determined that the detected delay timeexceeds the predetermined value (such as 500 ms), (“YES” at S211), atS212, the delay detector 18 requests the imaging unit 14 to stopcapturing the image of the object such as the user.

At S213, the imaging unit 14 stops capturing the image such that imagedata is not transmitted from the imaging unit 14 to the datatransmitter/receiver 11. With no image data being received, the datatransmitter/receiver 11 stops transmission of image data through thecommunication network 2, such that only sound data is transmitted fromthe terminal 10 db to the terminal 10 aa.

When it is determined that the detected delay time does not exceed thepredetermined value (“NO” at S211), the operation of FIG. 39 endswithout performing S212 and S213, as there is no need to interruptcapturing of image data to resolve the problem in delay in datatransmission. In such case, the contents data including image data andsound data is transmitted to the terminal 10 aa via the relay device 30a.

In the above-described example referring to FIG. 39, interruption ofimage capturing can further reduce energy consumption by the terminal10, thus saving energy.

In case the data transfer capability is lowered, for example, due to theincreased load on the communication network 2, the terminal 10 db totransmit image data simply stops transmission of image data. Thus, thereis no need to request the counterpart terminal 10 aa to interrupttransmission of image data, for example, via the management system 50,or there is no need for the management system 50 to manage variousinformation such as the delay time information or the image stateinformation. This easily reduces the data traffic as no image data istransmitted, thus contributing to the solution to the increased datatraffic on the communication network. Further, as the sound data istransmitted to the counterpart terminal 10 aa, the users are able tocommunicate with each other to carry out teleconference.

Further, since the terminal 10 db stops transmission of image data, theterminal 10 db does not have to send a request to the counterpartterminal 10 aa to interrupt transmission of image data from thecounterpart terminal 10 aa to the terminal 10 db. In one example, whenthe counterpart terminal 10 aa detects a delay in reception of contentsdata, the terminal 10 aa performs operation of FIG. 38 in asubstantially similar manner as described above, thus transmitting onlysound data. In another example, when the counterpart terminal 10 aa doesnot detect a delay in reception of contents data, the terminal 10 aacontinues to transmit contents data including image data and sound data.

In any one of the above-described examples, the relay device 30, themanagement system 50, the program providing system 90, and themaintenance system 100 may be each implemented by a single computer.Alternatively, any number of parts, functions, or modules of the relaydevice 30, the management system 50, the program providing system 90,and the maintenance system 100 may be classified into a desired numberof groups to be carried out by a plurality of computers. In case theprogram providing system 90 is implemented by the single computer, theprogram to be provided by the program providing system 90 may betransmitted, one module by one module, after dividing into a pluralityof modules, or may be transmitted at once. In case the program providingsystem 90 is implemented as a plurality of computers, each computer maytransmit each module that is stored in its memory, after the program isdivided into a plurality of modules.

A recording medium storing any one of the terminal control program,relay control program, and transmission management program, or a storagedevice such as the HDD 204 that stores any one of the terminal controlprogram, relay control program, and transmission management program, orthe program providing system 90 provided with the HD 204 storing any oneof the terminal control program, relay control program, and transmissionmanagement program, may be distributed within the country or to anothercountry as a computer program product.

In the above-described examples, the quality of image data to beprocessed by the relay device 30, which is determined based oninformation obtainable from any one of the image state change managementtable of FIG. 11 and the image state management table of FIG. 19, isanalyzed in terms of image resolution. Alternatively, any other criteriamay be used to analyze quality of image data including, for example,depth of image, sampling frequency in case of sound data, and bit lengthin case of sound data. Further, the sound data may be transmitted orreceived in three items of sound data including high-resolution sounddata, medium-resolution sound data, and low-resolution sound data.

Further, the date and time information stored in the relay devicemanagement table of FIG. 12 or the terminal management table of FIG. 14,or the delay information stored in the session management table of FIG.16, is expressed in terms of date and time. Alternatively, the date andtime information or the delay information may be expressed only in termsof time such as the time at which information is received.

Further, in the above-described examples, the relay device IP address ofthe relay device 30 and the terminal IP address of the terminal 10 arerespectively managed using the relay device management table of FIG. 12and the terminal management table of FIG. 14. Alternatively, the relaydevice 30 and the terminal 10 may each be managed using any otheridentification information or using any other tables. For example, whenthe relay device 30 or the terminal 10 needs to be identified on thecommunication network 2, the relay device 30 or the terminal 10 may bemanaged using Fully Qualified Domain Name (FQDN). In such case, thetransmission system 10 is provided with a domain name system (DNS)server that obtains the IP address that corresponds to the FQDN of therelay device 30 or the terminal 10. In view of this, identificationinformation for identifying the relay device 30 on the communicationnetwork 2 may not only include the identification information thatidentifies the relay device 30 on the communication network 2, but alsoidentification information that identifies a node on the communicationnetwork 2 to which the relay device 30 is connected, or identificationinformation that identifies a node on the communication network 2 fromwhich the relay device 30 is connected. Similarly, identificationinformation for identifying the terminal 10 on the communication network2 may not only include the identification information that identifiesthe terminal 10 on the communication network 2, but also identificationinformation that identifies a node on the communication network 2 towhich the terminal 10 is connected, or identification information thatidentifies a node on the communication network 2 from which the terminal10 is connected.

In the above-described examples, the transmission system 1 of FIG. 1 istreated as a videoconference system. Alternatively, the transmissionsystem 1 of FIG. 1 may be implemented as a teleconference system such asthe IP teleconference system or the Internet teleconference system.Alternatively, the transmission system 1 of FIG. 1 may be implemented asa car navigation system. For example, the request terminal 10 may beimplemented as a car navigation system that is installed onto anautomobile. The counterpart terminal 10 may be implemented as amanagement terminal or server at a management center that manages thecar navigation system or a car navigation system that is installed ontoanother automobile.

In another example, the transmission system 1 of FIG. 1 may beimplemented as a communication system having a portable phone. In suchcase, the terminal 10 is implemented as the portable phone. The terminal10, or the portable phone 10, includes a body, a menu screen displaybutton, a display section, a microphone provided at a lower portion ofthe body, and a speaker provided at an upper portion of the body. Whenselected, the menu screen display button causes the display section todisplay a menu screen in which various icons each indicating a specificapplication program are displayed. In this example, the display sectiondisplays a candidate terminal list that lists a plurality of terminalnames together with a plurality of icons each reflecting the operationstate of each candidate terminal. Since the terminal 10 in this exampleis implemented as a portable phone, the name of a user who owns thespecific terminal, or a nickname of the user, is displayed as theterminal name. The display section is a touch panel screen, which allowsthe user to select one of the plurality of terminal names beingdisplayed by the display section. When a specific terminal name, or auser name, is selected, the portable phone starts communication with thespecific terminal that is selected in a substantially similar manner asdescribed above.

In the above-described examples, the contents data is assumed to includeimage data and sound data. Alternatively, the contents data may includeany other type of data that affects human senses of sight in alternativeto image data, or any other type of data that affects human senses ofhearing in alternative to sound data. Alternatively, the contents datamay include any other type of data that affects human senses of sight,smell, taste, touch, and hearing. In case the contents data that affectshuman senses of touch, the terminal 10 may convey the contents data thatreflects senses of touch that is felt by a user at the terminal 10 toanother terminal 10 through the communication network 2. In case thecontents data that affects human senses of smell, the terminal 10 mayconvey the contents data that affects senses of smell felt by a user atthe terminal 10 to another terminal 10 through the communication network2. In case the contents data that affects human senses of taste, theterminal 10 may convey the contents data that affects senses of tastefelt by a user at the terminal 10 to another terminal 10 through thecommunication network 2.

Further, the contents data may only include one type of contents dataselected from sight data such as image data, hearing data such as sounddata, touch data, smell data, and taste data.

In case the contents data includes data other than the image data andthe sound data, the transmission terminal 10 may transmit the contentsdata including at least the sound data, but excluding the image data, tothe counterpart terminal 10, when the delay time exceeds thepredetermined value.

Further, in the above-described examples, the transmissions system 1 isimplemented as a videoconference system for use at offices. Otherexamples of use of the transmission system 1 include, but not limitedto, meetings, casual conversation among family members or friends, anddistribution of information in one direction.

Numerous additional modifications and variations are possible in lightof the above teachings. It is therefore to be understood that within thescope of the appended claims, the disclosure of the present inventionmay be practiced otherwise than as specifically described herein. Forexample, elements and/or features of different illustrative embodimentsmay be combined with each other and/or substituted for each other withinthe scope of this disclosure and appended claims.

Further, any of the above-described devices or units can be implementedas a hardware apparatus, such as a special-purpose circuit or device, oras a hardware/software combination, such as a processor executing asoftware program. Further, as described above, any one of theabove-described and other methods of the present invention may beembodied in the form of a computer program stored in any kind of storagemedium. Examples of storage mediums include, but are not limited to,flexible disk, hard disk, optical discs, magneto-optical discs, magnetictapes, nonvolatile memory cards, ROM (read-only-memory), etc.

Alternatively, any one of the above-described and other methods of thepresent invention may be implemented by ASIC, prepared byinterconnecting an appropriate network of conventional componentcircuits or by a combination thereof with one or more conventionalgeneral purpose microprocessors and/or signal processors programmedaccordingly.

The present invention can be implemented in any convenient form, forexample using dedicated hardware, or a mixture of dedicated hardware andsoftware. The present invention may be implemented as computer softwareimplemented by one or more networked processing apparatuses. The networkcan comprise any conventional terrestrial or wireless communicationsnetwork, such as the Internet. The processing apparatuses can compromiseany suitably programmed apparatuses such as a general purpose computer,personal digital assistant, mobile telephone (such as a WAP or3G-compliant phone) and so on. Since the present invention can beimplemented as software, each and every aspect of the present inventionthus encompasses computer software implementable on a programmabledevice. The computer software can be provided to the programmable deviceusing any storage medium for storing processor readable code such as afloppy disk, hard disk, CD ROM, magnetic tape device or solid statememory device.

The hardware platform includes any desired kind of hardware resourcesincluding, for example, a central processing unit (CPU), a random accessmemory (RAM), and a hard disk drive (HDD). The CPU may be implemented byany desired kind of any desired number of processor. The RAM may beimplemented by any desired kind of volatile or non-volatile memory. TheHDD may be implemented by any desired kind of non-volatile memorycapable of storing a large amount of data. The hardware resources mayadditionally include an input device, an output device, or a networkdevice, depending on the type of the apparatus. Alternatively, the HDDmay be provided outside of the apparatus as long as the HDD isaccessible. In this example, the CPU, such as a cashe memory of the CPU,and the RAM may function as a physical memory or a primary memory of theapparatus, while the HDD may function as a secondary memory of theapparatus.

In one example, the present invention may reside in a transmissionterminal that communicates contents data including sound data with acounterpart transmission terminal. The transmission terminal includes:means for receiving contents data from the counterpart transmissionterminal; means for transmitting contents data to the counterparttransmission terminal; and means for detecting a delay time indicating adelay in receiving the contents data from the counterpart transmissionterminal. When the delay time exceeds a predetermined value, the meansfor detecting causes the transmitting to interrupt transmission of thecontents data other than the sound data.

In the above-described example, the means for detecting causes the meansfor transmitting to interrupt transmission of image data included in thecontents data.

In the above-described example, the transmission terminal furtherincludes: means for capturing an image of an object at the transmissionterminal. The image captured by the means for capturing is included inthe contents data for transmission to the counterpart transmissionterminal.

In the above-described example, when the delay time exceeds thepredetermined value, the means for detecting causes the means forcapturing to interrupt capturing of the image of the object.

What is claimed is:
 1. A transmission terminal that communicates with acounterpart transmission terminal, the transmission terminal comprising:a network interface configured to receive first content data from thecounterpart transmission terminal, the first content data includingfirst image data and first sound data, and to transmit second contentdata to the counterpart transmission terminal, the second content dataincluding second image data and second sound data; and a processorconfigured to detect a delay time indicating a delay in receiving thefirst content data from the counterpart transmission terminal, wherein,upon the detected delay time exceeding a predetermined value, theprocessor causes an imaging unit to interrupt capturing of the image ofthe object and further causes the network interface to transmit, of thesecond content data, only the second sound data, to the counterparttransmission terminal.
 2. The transmission terminal of claim 1, whereinthe processor causes the network interface to interrupt transmission ofthe second image data included in the second content data.
 3. Thetransmission terminal of claim 2, further comprising: an imaging unitconfigured to capture an image of an object at the transmissionterminal, wherein the image captured by the imaging unit is included inthe second content data for transmission to the counterpart transmissionterminal.
 4. A method of controlling communication with a counterparttransmission terminal, performed by a transmission terminal, the methodcomprising: receiving first content data from the counterparttransmission terminal, the first content data including first image dataand first sound data; detecting a delay time indicating a delay inreceiving the first content data from the counterpart transmissionterminal; determining whether the detected delay time exceeds apredetermined value, wherein, upon determining that the detected delaytime exceeds the predetermined value, a processor causes an imaging unitto interrupt capturing of an image of an object; and transmitting, ofsecond content data, the second content data including second image dataand second sound data, only the second sound data, to the counterparttransmission terminal.
 5. The method of claim 4, wherein thetransmitting step comprises interrupting transmission of the secondimage data included in the second content data.
 6. The method of claim5, further comprising: capturing an image of an object at thetransmission terminal, wherein the captured image is included in thesecond content data for transmission to the counterpart transmissionterminal.
 7. A non-transitory recording medium storing a plurality ofinstructions which, when executed by a processor of a transmissionterminal, cause the processor to perform a method of controllingcommunication with a counterpart transmission terminal, the methodcomprising: receiving first content data from the counterparttransmission terminal, the first content data including first image dataand first sound data; detecting a delay time indicating a delay inreceiving the first content data from the counterpart transmissionterminal; determining whether the detected delay time exceeds apredetermined value, wherein upon determining that the detected delaytime exceeds the predetermined value, a processor causes the imagingunit to interrupt capturing of an image of an object; and transmitting,of second content data, the second content data including second imagedata and second sound data, only the second sound data, to thecounterpart transmission terminal.
 8. A transmission system, comprising:the transmission terminal of claim 1; and a management apparatusincluding processing circuitry configured to generate a messageindicating that the second image data was not received, when determiningthat the transmission terminal detected the delay time exceeding thepredetermined value, and transmit the generated message to thecounterpart transmission terminal for display.
 9. A transmissionmanagement system, comprising: a transmission terminal that communicateswith a counterpart terminal and includes first processing circuitryconfigured to detect a delay time in receiving content data from thecounterpart terminal, and a transmitter configured to transmit thedetected delay to a management apparatus; the management apparatusincluding a receiver to receive the detected delay, and secondprocessing circuitry configured to determine an image data quality usingthe received detected delay, and a transmitter configured to transmitthe determined image quality to a relay device; and a relay deviceincluding a receiver to receive the determined image data quality fromthe management apparatus and to receive content data from thecounterpart terminal, the content data including a plurality of contentdata at a corresponding plurality of quality levels, and thirdprocessing circuitry configured to determine which of the receivedplurality of content data to relay to the transmission terminal, basedon the received image quality data.